Oct 30 18:30:24.735724 ip-10-0-140-163 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Oct 30 18:30:24.735739 ip-10-0-140-163 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Oct 30 18:30:24.735748 ip-10-0-140-163 systemd[1]: kubelet.service: Failed with result 'resources'. Oct 30 18:30:24.736068 ip-10-0-140-163 systemd[1]: Failed to start Kubernetes Kubelet. Oct 30 18:30:35.961359 ip-10-0-140-163 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Oct 30 18:30:35.961377 ip-10-0-140-163 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 3b212ecd8fbd45c58b6eb3b18acb359d -- Oct 30 18:32:41.668459 ip-10-0-140-163 systemd[1]: Starting Kubernetes Kubelet... Oct 30 18:32:42.052914 ip-10-0-140-163 kubenswrapper[2575]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 18:32:42.052914 ip-10-0-140-163 kubenswrapper[2575]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 18:32:42.052914 ip-10-0-140-163 kubenswrapper[2575]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 18:32:42.052914 ip-10-0-140-163 kubenswrapper[2575]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 18:32:42.058370 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058286 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:32:42.058370 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058368 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:32:42.058370 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058372 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:32:42.058370 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058375 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:32:42.058531 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058379 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:32:42.058531 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058382 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:32:42.058531 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058384 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:32:42.058531 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058387 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:32:42.058531 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058390 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:32:42.058531 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058393 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:32:42.058531 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058395 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:32:42.058531 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058398 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:32:42.058531 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058401 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:32:42.058531 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058403 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:32:42.058531 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058406 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:32:42.058531 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058409 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:32:42.058531 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058411 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:32:42.058531 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058415 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:32:42.058531 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058418 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:32:42.058531 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058421 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:32:42.058531 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058423 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:32:42.058531 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058426 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:32:42.058531 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058429 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:32:42.058531 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058431 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:32:42.058992 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058433 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:32:42.058992 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058436 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:32:42.058992 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058439 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:32:42.058992 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058441 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:32:42.058992 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058444 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:32:42.058992 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058447 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:32:42.058992 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058449 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:32:42.058992 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058459 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:32:42.058992 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058463 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:32:42.058992 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058466 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:32:42.058992 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058468 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:32:42.058992 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058471 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:32:42.058992 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058474 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:32:42.058992 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058477 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:32:42.058992 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058479 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:32:42.058992 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058482 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:32:42.058992 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058484 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:32:42.058992 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058487 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:32:42.058992 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058490 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:32:42.059462 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058493 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:32:42.059462 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058495 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:32:42.059462 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058498 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:32:42.059462 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058500 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:32:42.059462 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058503 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:32:42.059462 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058505 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:32:42.059462 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058508 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:32:42.059462 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058510 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:32:42.059462 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058513 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:32:42.059462 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058516 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:32:42.059462 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058519 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:32:42.059462 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058521 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:32:42.059462 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058525 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:32:42.059462 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058529 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:32:42.059462 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058535 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:32:42.059462 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058538 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:32:42.059462 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058541 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:32:42.059462 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058545 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:32:42.059462 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058548 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:32:42.059462 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058550 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:32:42.059928 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058553 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:32:42.059928 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058556 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:32:42.059928 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058559 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:32:42.059928 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058562 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:32:42.059928 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058564 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:32:42.059928 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058567 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:32:42.059928 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058570 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:32:42.059928 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058573 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:32:42.059928 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058576 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:32:42.059928 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058579 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:32:42.059928 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058582 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:32:42.059928 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058584 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:32:42.059928 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058587 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:32:42.059928 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058589 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:32:42.059928 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058592 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:32:42.059928 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058596 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:32:42.059928 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058601 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:32:42.059928 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058604 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:32:42.059928 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058607 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:32:42.060392 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058610 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:32:42.060392 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058614 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:32:42.060392 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058617 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:32:42.060392 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058620 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:32:42.060392 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058622 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:32:42.060392 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058625 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:32:42.060392 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058628 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:32:42.060392 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058630 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:32:42.060392 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058633 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:32:42.060392 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058635 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:32:42.060392 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058638 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:32:42.060392 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058642 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:32:42.060392 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058644 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:32:42.060392 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058647 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:32:42.060392 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058650 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:32:42.060392 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058652 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:32:42.060392 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058655 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:32:42.060392 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058658 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:32:42.060392 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058661 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:32:42.060887 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058664 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:32:42.060887 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058666 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:32:42.060887 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058669 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:32:42.060887 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058672 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:32:42.060887 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.058674 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:32:42.060887 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059477 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:32:42.060887 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059485 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:32:42.060887 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059488 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:32:42.060887 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059490 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:32:42.060887 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059493 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:32:42.060887 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059496 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:32:42.060887 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059499 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:32:42.060887 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059502 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:32:42.060887 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059505 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:32:42.060887 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059508 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:32:42.060887 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059512 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:32:42.060887 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059515 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:32:42.060887 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059518 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:32:42.060887 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059520 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:32:42.061360 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059524 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:32:42.061360 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059526 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:32:42.061360 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059529 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:32:42.061360 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059532 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:32:42.061360 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059534 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:32:42.061360 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059537 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:32:42.061360 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059539 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:32:42.061360 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059542 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:32:42.061360 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059545 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:32:42.061360 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059548 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:32:42.061360 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059551 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:32:42.061360 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059553 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:32:42.061360 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059556 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:32:42.061360 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059560 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:32:42.061360 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059562 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:32:42.061360 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059565 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:32:42.061360 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059568 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:32:42.061360 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059570 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:32:42.061360 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059573 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:32:42.061360 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059576 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:32:42.061837 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059578 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:32:42.061837 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059581 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:32:42.061837 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059583 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:32:42.061837 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059586 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:32:42.061837 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059588 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:32:42.061837 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059591 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:32:42.061837 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059593 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:32:42.061837 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059596 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:32:42.061837 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059598 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:32:42.061837 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059602 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:32:42.061837 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059605 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:32:42.061837 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059607 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:32:42.061837 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059610 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:32:42.061837 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059612 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:32:42.061837 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059615 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:32:42.061837 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059617 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:32:42.061837 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059620 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:32:42.061837 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059622 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:32:42.061837 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059625 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:32:42.061837 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059627 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:32:42.061837 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059630 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:32:42.062348 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059634 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:32:42.062348 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059636 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:32:42.062348 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059639 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:32:42.062348 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059642 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:32:42.062348 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059645 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:32:42.062348 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059647 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:32:42.062348 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059650 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:32:42.062348 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059652 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:32:42.062348 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059655 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:32:42.062348 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059659 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:32:42.062348 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059661 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:32:42.062348 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059664 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:32:42.062348 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059666 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:32:42.062348 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059669 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:32:42.062348 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059671 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:32:42.062348 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059674 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:32:42.062348 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059676 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:32:42.062348 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059679 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:32:42.062348 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059682 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:32:42.062801 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059684 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:32:42.062801 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059687 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:32:42.062801 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059691 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:32:42.062801 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059695 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:32:42.062801 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059698 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:32:42.062801 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059701 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:32:42.062801 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059704 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:32:42.062801 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059707 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:32:42.062801 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059709 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:32:42.062801 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059712 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:32:42.062801 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059714 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:32:42.062801 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059717 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:32:42.062801 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059719 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:32:42.062801 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059722 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:32:42.062801 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059725 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:32:42.062801 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059728 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:32:42.062801 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059731 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:32:42.062801 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059734 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:32:42.063262 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059737 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:32:42.063262 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059739 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:32:42.063262 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059742 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:32:42.063262 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059745 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:32:42.063262 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059748 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:32:42.063262 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059750 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:32:42.063262 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059753 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:32:42.063262 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059755 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:32:42.063262 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059758 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:32:42.063262 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059763 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:32:42.063262 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059766 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:32:42.063262 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059769 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:32:42.063262 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059772 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:32:42.063262 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.059775 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:32:42.063262 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060277 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:32:42.063262 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060284 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:32:42.063262 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060288 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:32:42.063262 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060291 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:32:42.063262 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060294 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:32:42.063262 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060296 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:32:42.063728 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060299 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:32:42.063728 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060302 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:32:42.063728 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060304 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:32:42.063728 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060307 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:32:42.063728 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060310 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:32:42.063728 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060313 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:32:42.063728 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060316 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:32:42.063728 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060318 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:32:42.063728 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060321 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:32:42.063728 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060324 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:32:42.063728 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060327 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:32:42.063728 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060330 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:32:42.063728 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060332 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:32:42.063728 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060335 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:32:42.063728 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060338 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:32:42.063728 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060340 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:32:42.063728 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060343 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:32:42.063728 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060346 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:32:42.063728 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060349 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:32:42.064196 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060353 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:32:42.064196 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060356 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:32:42.064196 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060359 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:32:42.064196 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060362 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:32:42.064196 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060364 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:32:42.064196 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060367 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:32:42.064196 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060370 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:32:42.064196 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060373 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:32:42.064196 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060375 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:32:42.064196 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060378 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:32:42.064196 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060381 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:32:42.064196 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060384 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:32:42.064196 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060386 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:32:42.064196 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060389 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:32:42.064196 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060391 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:32:42.064196 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060397 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:32:42.064196 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060400 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:32:42.064196 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060403 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:32:42.064196 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060406 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:32:42.064196 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060409 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:32:42.064676 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060411 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:32:42.064676 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060414 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:32:42.064676 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060417 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:32:42.064676 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060420 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:32:42.064676 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060422 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:32:42.064676 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060426 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:32:42.064676 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060428 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:32:42.064676 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060431 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:32:42.064676 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060434 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:32:42.064676 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060436 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:32:42.064676 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060439 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:32:42.064676 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060441 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:32:42.064676 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060444 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:32:42.064676 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060446 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:32:42.064676 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060449 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:32:42.064676 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060452 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:32:42.064676 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060454 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:32:42.064676 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060457 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:32:42.064676 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060460 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:32:42.064676 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060462 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:32:42.065162 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060465 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:32:42.065162 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060467 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:32:42.065162 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060470 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:32:42.065162 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060473 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:32:42.065162 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060476 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:32:42.065162 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060478 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:32:42.065162 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060481 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:32:42.065162 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060484 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:32:42.065162 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060488 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:32:42.065162 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060493 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:32:42.065162 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060498 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:32:42.065162 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060501 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:32:42.065162 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060504 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:32:42.065162 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060507 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:32:42.065162 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060510 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:32:42.065162 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060512 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:32:42.065162 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060515 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:32:42.065162 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060518 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:32:42.065162 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060520 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:32:42.065652 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060523 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:32:42.065652 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060525 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:32:42.065652 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060528 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:32:42.065652 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060530 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:32:42.065652 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060533 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:32:42.065652 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060535 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:32:42.065652 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060538 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:32:42.065652 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060541 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:32:42.065652 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060543 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:32:42.065652 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060546 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:32:42.065652 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060549 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:32:42.065652 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060551 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:32:42.065652 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060553 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:32:42.065652 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060556 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:32:42.065652 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060558 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:32:42.065652 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060561 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:32:42.065652 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060564 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:32:42.065652 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060567 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:32:42.065652 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060570 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:32:42.066105 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060573 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:32:42.066105 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060576 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:32:42.066105 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.060579 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:32:42.066105 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.060666 2575 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 18:32:42.066105 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.060674 2575 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 18:32:42.066105 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.060683 2575 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 18:32:42.066105 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.060688 2575 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 18:32:42.066105 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.060692 2575 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 18:32:42.066105 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.060695 2575 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 18:32:42.066105 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.060700 2575 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 18:32:42.066105 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.060704 2575 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 18:32:42.066105 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.060708 2575 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 18:32:42.066105 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.060711 2575 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 18:32:42.066105 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.060715 2575 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 18:32:42.066105 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.060719 2575 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 18:32:42.066105 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.060722 2575 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 18:32:42.066105 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.060725 2575 flags.go:64] FLAG: --cgroup-root="" Oct 30 18:32:42.066105 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.060737 2575 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 18:32:42.066105 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.060741 2575 flags.go:64] FLAG: --client-ca-file="" Oct 30 18:32:42.066105 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.060744 2575 flags.go:64] FLAG: --cloud-provider="external" Oct 30 18:32:42.066105 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.060747 2575 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 18:32:42.066105 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.060751 2575 flags.go:64] FLAG: --cluster-domain="" Oct 30 18:32:42.066105 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.060754 2575 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 18:32:42.066105 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.060757 2575 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 18:32:42.066694 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.060760 2575 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 18:32:42.066694 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.060764 2575 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 18:32:42.066694 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.060772 2575 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 18:32:42.066694 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.060775 2575 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 18:32:42.066694 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.060778 2575 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 18:32:42.066694 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.060782 2575 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 18:32:42.066694 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.060785 2575 flags.go:64] FLAG: --contention-profiling="false" Oct 30 18:32:42.066694 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.060789 2575 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 18:32:42.066694 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.060792 2575 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 18:32:42.066694 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.060795 2575 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 18:32:42.066694 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.060798 2575 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 18:32:42.066694 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.060802 2575 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 18:32:42.066694 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.060807 2575 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 18:32:42.066694 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.060810 2575 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 18:32:42.066694 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.060813 2575 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 18:32:42.066694 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.060816 2575 flags.go:64] FLAG: --enable-server="true" Oct 30 18:32:42.066694 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.060819 2575 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 18:32:42.066694 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.060823 2575 flags.go:64] FLAG: --event-burst="100" Oct 30 18:32:42.066694 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.060826 2575 flags.go:64] FLAG: --event-qps="50" Oct 30 18:32:42.066694 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.060829 2575 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 18:32:42.066694 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.060832 2575 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 18:32:42.066694 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.060835 2575 flags.go:64] FLAG: --eviction-hard="" Oct 30 18:32:42.066694 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.060839 2575 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 18:32:42.066694 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.060842 2575 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 18:32:42.067307 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.060845 2575 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 18:32:42.067307 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.060848 2575 flags.go:64] FLAG: --eviction-soft="" Oct 30 18:32:42.067307 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.060851 2575 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 18:32:42.067307 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.060854 2575 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 18:32:42.067307 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.060857 2575 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 18:32:42.067307 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.060860 2575 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 18:32:42.067307 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.060863 2575 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 18:32:42.067307 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.060866 2575 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 18:32:42.067307 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.060869 2575 flags.go:64] FLAG: --feature-gates="" Oct 30 18:32:42.067307 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.060873 2575 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 18:32:42.067307 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.060877 2575 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 18:32:42.067307 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.060880 2575 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 18:32:42.067307 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.060883 2575 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 18:32:42.067307 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.060886 2575 flags.go:64] FLAG: --healthz-port="10248" Oct 30 18:32:42.067307 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.060889 2575 flags.go:64] FLAG: --help="false" Oct 30 18:32:42.067307 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.060892 2575 flags.go:64] FLAG: --hostname-override="ip-10-0-140-163.us-east-2.compute.internal" Oct 30 18:32:42.067307 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.060911 2575 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 18:32:42.067307 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.060915 2575 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 18:32:42.067307 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.060918 2575 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 18:32:42.067307 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.060922 2575 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 18:32:42.067307 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.060927 2575 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 18:32:42.067307 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.060930 2575 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 18:32:42.067307 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.060933 2575 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 18:32:42.067888 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.060936 2575 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 18:32:42.067888 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.060939 2575 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 18:32:42.067888 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.060942 2575 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 18:32:42.067888 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.060945 2575 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 18:32:42.067888 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.060948 2575 flags.go:64] FLAG: --kube-reserved="" Oct 30 18:32:42.067888 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.060951 2575 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 18:32:42.067888 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.060954 2575 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 18:32:42.067888 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.060957 2575 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 18:32:42.067888 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.060960 2575 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 18:32:42.067888 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.060962 2575 flags.go:64] FLAG: --lock-file="" Oct 30 18:32:42.067888 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.060965 2575 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 18:32:42.067888 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.060968 2575 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 18:32:42.067888 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.060971 2575 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 18:32:42.067888 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.060977 2575 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 18:32:42.067888 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.060980 2575 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 18:32:42.067888 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.060983 2575 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 18:32:42.067888 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.060985 2575 flags.go:64] FLAG: --logging-format="text" Oct 30 18:32:42.067888 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.060988 2575 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 18:32:42.067888 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.060992 2575 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 18:32:42.067888 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.060994 2575 flags.go:64] FLAG: --manifest-url="" Oct 30 18:32:42.067888 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.060997 2575 flags.go:64] FLAG: --manifest-url-header="" Oct 30 18:32:42.067888 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.061006 2575 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 18:32:42.067888 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.061009 2575 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 18:32:42.067888 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.061013 2575 flags.go:64] FLAG: --max-pods="110" Oct 30 18:32:42.067888 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.061016 2575 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 18:32:42.068625 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.061019 2575 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 18:32:42.068625 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.061024 2575 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 18:32:42.068625 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.061027 2575 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 18:32:42.068625 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.061030 2575 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 18:32:42.068625 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.061035 2575 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 18:32:42.068625 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.061037 2575 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 18:32:42.068625 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.061044 2575 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 18:32:42.068625 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.061047 2575 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 18:32:42.068625 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.061051 2575 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 18:32:42.068625 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.061054 2575 flags.go:64] FLAG: --pod-cidr="" Oct 30 18:32:42.068625 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.061056 2575 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 18:32:42.068625 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.061059 2575 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 18:32:42.068625 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.061063 2575 flags.go:64] FLAG: --pods-per-core="0" Oct 30 18:32:42.068625 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.061066 2575 flags.go:64] FLAG: --port="10250" Oct 30 18:32:42.068625 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.061068 2575 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 18:32:42.068625 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.061071 2575 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0a85cc7f9f2bfb89b" Oct 30 18:32:42.068625 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.061074 2575 flags.go:64] FLAG: --qos-reserved="" Oct 30 18:32:42.068625 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.061078 2575 flags.go:64] FLAG: --read-only-port="10255" Oct 30 18:32:42.068625 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.061081 2575 flags.go:64] FLAG: --register-node="true" Oct 30 18:32:42.068625 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.061084 2575 flags.go:64] FLAG: --register-with-taints="" Oct 30 18:32:42.068625 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.061087 2575 flags.go:64] FLAG: --registry-burst="10" Oct 30 18:32:42.068625 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.061090 2575 flags.go:64] FLAG: --registry-qps="5" Oct 30 18:32:42.068625 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.061093 2575 flags.go:64] FLAG: --reserved-cpus="" Oct 30 18:32:42.068625 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.061096 2575 flags.go:64] FLAG: --reserved-memory="" Oct 30 18:32:42.068625 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.061100 2575 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 18:32:42.068625 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.061103 2575 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 18:32:42.069306 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.061106 2575 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 18:32:42.069306 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.061109 2575 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 18:32:42.069306 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.061112 2575 flags.go:64] FLAG: --runonce="false" Oct 30 18:32:42.069306 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.061114 2575 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 18:32:42.069306 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.061118 2575 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 18:32:42.069306 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.061121 2575 flags.go:64] FLAG: --seccomp-default="false" Oct 30 18:32:42.069306 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.061123 2575 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 18:32:42.069306 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.061128 2575 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 18:32:42.069306 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.061131 2575 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 18:32:42.069306 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.061134 2575 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 18:32:42.069306 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.061152 2575 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 18:32:42.069306 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.061158 2575 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 18:32:42.069306 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.061162 2575 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 18:32:42.069306 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.061166 2575 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 18:32:42.069306 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.061169 2575 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 18:32:42.069306 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.061172 2575 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 18:32:42.069306 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.061175 2575 flags.go:64] FLAG: --system-cgroups="" Oct 30 18:32:42.069306 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.061178 2575 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 18:32:42.069306 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.061183 2575 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 18:32:42.069306 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.061186 2575 flags.go:64] FLAG: --tls-cert-file="" Oct 30 18:32:42.069306 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.061189 2575 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 18:32:42.069306 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.061193 2575 flags.go:64] FLAG: --tls-min-version="" Oct 30 18:32:42.069306 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.061196 2575 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 18:32:42.069306 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.061199 2575 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 18:32:42.069306 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.061202 2575 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 18:32:42.069907 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.061205 2575 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 18:32:42.069907 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.061208 2575 flags.go:64] FLAG: --v="2" Oct 30 18:32:42.069907 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.061213 2575 flags.go:64] FLAG: --version="false" Oct 30 18:32:42.069907 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.061217 2575 flags.go:64] FLAG: --vmodule="" Oct 30 18:32:42.069907 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.061221 2575 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 18:32:42.069907 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.061224 2575 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 18:32:42.069907 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061356 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:32:42.069907 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061359 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:32:42.069907 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061363 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:32:42.069907 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061366 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:32:42.069907 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061370 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:32:42.069907 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061372 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:32:42.069907 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061375 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:32:42.069907 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061378 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:32:42.069907 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061382 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:32:42.069907 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061384 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:32:42.069907 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061388 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:32:42.069907 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061391 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:32:42.069907 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061394 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:32:42.069907 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061396 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:32:42.069907 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061399 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:32:42.070435 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061401 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:32:42.070435 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061404 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:32:42.070435 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061406 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:32:42.070435 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061409 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:32:42.070435 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061411 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:32:42.070435 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061414 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:32:42.070435 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061416 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:32:42.070435 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061419 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:32:42.070435 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061421 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:32:42.070435 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061424 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:32:42.070435 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061427 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:32:42.070435 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061430 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:32:42.070435 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061432 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:32:42.070435 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061435 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:32:42.070435 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061438 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:32:42.070435 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061440 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:32:42.070435 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061443 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:32:42.070435 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061445 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:32:42.070435 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061448 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:32:42.070875 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061450 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:32:42.070875 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061453 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:32:42.070875 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061456 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:32:42.070875 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061460 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:32:42.070875 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061463 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:32:42.070875 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061466 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:32:42.070875 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061470 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:32:42.070875 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061472 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:32:42.070875 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061475 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:32:42.070875 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061478 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:32:42.070875 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061481 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:32:42.070875 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061483 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:32:42.070875 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061486 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:32:42.070875 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061489 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:32:42.070875 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061491 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:32:42.070875 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061494 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:32:42.070875 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061496 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:32:42.070875 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061499 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:32:42.070875 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061502 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:32:42.070875 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061504 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:32:42.071381 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061507 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:32:42.071381 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061509 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:32:42.071381 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061512 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:32:42.071381 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061514 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:32:42.071381 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061517 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:32:42.071381 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061520 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:32:42.071381 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061524 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:32:42.071381 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061527 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:32:42.071381 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061530 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:32:42.071381 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061533 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:32:42.071381 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061536 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:32:42.071381 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061539 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:32:42.071381 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061541 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:32:42.071381 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061544 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:32:42.071381 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061546 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:32:42.071381 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061549 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:32:42.071381 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061552 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:32:42.071381 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061554 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:32:42.071381 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061558 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:32:42.071849 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061560 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:32:42.071849 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061563 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:32:42.071849 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061566 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:32:42.071849 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061569 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:32:42.071849 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061571 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:32:42.071849 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061574 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:32:42.071849 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061576 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:32:42.071849 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061579 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:32:42.071849 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061582 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:32:42.071849 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061584 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:32:42.071849 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061587 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:32:42.071849 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061589 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:32:42.071849 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061592 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:32:42.071849 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061594 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:32:42.071849 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061597 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:32:42.071849 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061599 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:32:42.071849 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061602 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:32:42.071849 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061604 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:32:42.071849 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061607 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:32:42.071849 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061609 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:32:42.072344 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061612 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:32:42.072344 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061615 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:32:42.072344 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061617 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:32:42.072344 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061620 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:32:42.072344 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061622 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:32:42.072344 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061625 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:32:42.072344 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061627 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:32:42.072344 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061630 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:32:42.072344 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061632 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:32:42.072344 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061635 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:32:42.072344 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061637 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:32:42.072344 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061641 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:32:42.072344 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061644 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:32:42.072344 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.061669 2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 18:32:42.072344 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061705 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:32:42.072344 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061709 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:32:42.072736 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061712 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:32:42.072736 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061715 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:32:42.072736 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061718 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:32:42.072736 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061721 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:32:42.072736 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061723 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:32:42.072736 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061726 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:32:42.072736 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061729 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:32:42.072736 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061731 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:32:42.072736 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061734 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:32:42.072736 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061736 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:32:42.072736 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061740 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:32:42.072736 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061744 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:32:42.072736 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061746 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:32:42.072736 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061749 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:32:42.072736 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061752 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:32:42.072736 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061754 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:32:42.072736 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061757 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:32:42.072736 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061759 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:32:42.072736 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061762 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:32:42.073194 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061764 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:32:42.073194 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061767 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:32:42.073194 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061770 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:32:42.073194 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061772 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:32:42.073194 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061775 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:32:42.073194 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061777 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:32:42.073194 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061780 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:32:42.073194 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061782 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:32:42.073194 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061787 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:32:42.073194 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061790 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:32:42.073194 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061793 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:32:42.073194 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061796 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:32:42.073194 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061798 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:32:42.073194 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061801 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:32:42.073194 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061805 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:32:42.073194 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061807 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:32:42.073194 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061810 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:32:42.073194 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061812 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:32:42.073194 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061815 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:32:42.073194 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061818 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:32:42.073668 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061821 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:32:42.073668 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061823 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:32:42.073668 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061826 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:32:42.073668 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061828 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:32:42.073668 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061831 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:32:42.073668 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061834 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:32:42.073668 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061836 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:32:42.073668 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061839 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:32:42.073668 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061842 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:32:42.073668 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061844 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:32:42.073668 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061847 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:32:42.073668 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061850 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:32:42.073668 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061852 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:32:42.073668 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061855 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:32:42.073668 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061857 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:32:42.073668 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061860 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:32:42.073668 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061863 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:32:42.073668 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061865 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:32:42.073668 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061868 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:32:42.073668 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061871 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:32:42.074189 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061876 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:32:42.074189 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061879 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:32:42.074189 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061882 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:32:42.074189 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061884 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:32:42.074189 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061887 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:32:42.074189 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061889 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:32:42.074189 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061892 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:32:42.074189 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061895 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:32:42.074189 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061898 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:32:42.074189 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061901 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:32:42.074189 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061904 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:32:42.074189 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061907 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:32:42.074189 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061909 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:32:42.074189 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061912 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:32:42.074189 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061914 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:32:42.074189 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061917 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:32:42.074189 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061919 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:32:42.074189 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061922 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:32:42.074189 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061925 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:32:42.074668 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061927 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:32:42.074668 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061930 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:32:42.074668 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061932 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:32:42.074668 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061935 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:32:42.074668 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061938 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:32:42.074668 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061940 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:32:42.074668 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061943 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:32:42.074668 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061946 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:32:42.074668 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061948 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:32:42.074668 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061951 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:32:42.074668 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061953 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:32:42.074668 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061956 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:32:42.074668 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061959 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:32:42.074668 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061964 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:32:42.074668 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061966 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:32:42.074668 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061970 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:32:42.074668 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061975 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:32:42.074668 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061978 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:32:42.074668 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061982 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:32:42.075110 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061985 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:32:42.075110 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061988 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:32:42.075110 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061991 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:32:42.075110 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061994 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:32:42.075110 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061996 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:32:42.075110 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.061999 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:32:42.075110 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.062001 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:32:42.075110 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.069633 2575 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 18:32:42.075110 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.069648 2575 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 18:32:42.075110 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.069662 2575 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 18:32:42.075110 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.069667 2575 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 18:32:42.075110 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.069737 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:32:42.075110 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.069741 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:32:42.075110 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.069745 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:32:42.075110 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.069748 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:32:42.075110 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.069751 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:32:42.075110 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.069754 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:32:42.075110 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.069757 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:32:42.075110 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.069760 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:32:42.075110 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.069762 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:32:42.075615 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.069766 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:32:42.075615 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.069768 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:32:42.075615 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.069771 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:32:42.075615 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.069773 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:32:42.075615 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.069776 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:32:42.075615 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.069779 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:32:42.075615 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.069782 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:32:42.075615 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.069784 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:32:42.075615 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.069787 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:32:42.075615 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.069789 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:32:42.075615 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.069792 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:32:42.075615 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.069794 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:32:42.075615 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.069797 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:32:42.075615 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.069799 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:32:42.075615 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.069802 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:32:42.075615 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.069805 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:32:42.075615 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.069807 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:32:42.075615 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.069810 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:32:42.075615 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.069812 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:32:42.075615 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.069815 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:32:42.076077 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.069819 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:32:42.076077 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.069822 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:32:42.076077 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.069825 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:32:42.076077 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.069827 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:32:42.076077 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.069831 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:32:42.076077 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.069836 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:32:42.076077 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.069839 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:32:42.076077 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.069842 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:32:42.076077 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.069844 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:32:42.076077 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.069847 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:32:42.076077 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.069850 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:32:42.076077 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.069852 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:32:42.076077 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.069855 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:32:42.076077 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.069857 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:32:42.076077 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.069860 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:32:42.076077 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.069862 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:32:42.076077 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.069865 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:32:42.076077 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.069868 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:32:42.076077 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.069870 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:32:42.076544 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.069873 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:32:42.076544 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.069875 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:32:42.076544 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.069878 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:32:42.076544 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.069880 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:32:42.076544 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.069883 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:32:42.076544 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.069887 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:32:42.076544 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.069890 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:32:42.076544 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.069893 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:32:42.076544 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.069897 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:32:42.076544 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.069899 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:32:42.076544 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.069902 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:32:42.076544 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.069905 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:32:42.076544 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.069907 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:32:42.076544 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.069910 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:32:42.076544 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.069914 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:32:42.076544 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.069917 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:32:42.076544 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.069920 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:32:42.076544 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.069923 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:32:42.076544 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.069925 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:32:42.076544 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.069928 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:32:42.077020 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.069931 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:32:42.077020 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.069933 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:32:42.077020 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.069936 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:32:42.077020 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.069938 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:32:42.077020 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.069941 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:32:42.077020 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.069943 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:32:42.077020 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.069946 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:32:42.077020 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.069948 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:32:42.077020 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.069951 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:32:42.077020 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.069953 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:32:42.077020 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.069956 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:32:42.077020 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.069958 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:32:42.077020 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.069961 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:32:42.077020 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.069963 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:32:42.077020 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.069966 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:32:42.077020 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.069968 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:32:42.077020 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.069971 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:32:42.077020 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.069973 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:32:42.077020 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.069976 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:32:42.077500 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.069978 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:32:42.077500 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.069981 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:32:42.077500 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.069983 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:32:42.077500 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.069986 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:32:42.077500 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.069988 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:32:42.077500 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.069991 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:32:42.077500 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.069994 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:32:42.077500 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.069996 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:32:42.077500 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.069999 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:32:42.077500 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070002 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:32:42.077500 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070004 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:32:42.077500 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070007 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:32:42.077500 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070009 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:32:42.077500 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070012 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:32:42.077500 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070015 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:32:42.077500 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070017 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:32:42.077500 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070020 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:32:42.077500 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070023 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:32:42.077500 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070025 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:32:42.077999 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.070051 2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 18:32:42.077999 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070198 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:32:42.077999 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070203 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:32:42.077999 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070206 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:32:42.077999 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070209 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:32:42.077999 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070212 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:32:42.077999 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070214 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:32:42.077999 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070217 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:32:42.077999 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070219 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:32:42.077999 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070222 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:32:42.077999 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070224 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:32:42.077999 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070227 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:32:42.077999 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070229 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:32:42.077999 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070232 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:32:42.077999 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070235 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:32:42.077999 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070237 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:32:42.078402 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070240 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:32:42.078402 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070242 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:32:42.078402 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070245 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:32:42.078402 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070248 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:32:42.078402 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070251 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:32:42.078402 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070253 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:32:42.078402 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070257 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:32:42.078402 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070259 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:32:42.078402 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070262 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:32:42.078402 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070264 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:32:42.078402 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070267 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:32:42.078402 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070269 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:32:42.078402 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070272 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:32:42.078402 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070275 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:32:42.078402 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070277 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:32:42.078402 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070280 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:32:42.078402 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070282 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:32:42.078402 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070285 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:32:42.078402 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070288 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:32:42.078402 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070290 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:32:42.079137 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070293 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:32:42.079137 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070295 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:32:42.079137 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070298 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:32:42.079137 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070300 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:32:42.079137 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070303 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:32:42.079137 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070306 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:32:42.079137 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070308 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:32:42.079137 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070310 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:32:42.079137 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070313 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:32:42.079137 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070316 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:32:42.079137 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070318 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:32:42.079137 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070321 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:32:42.079137 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070323 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:32:42.079137 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070326 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:32:42.079137 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070330 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:32:42.079137 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070332 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:32:42.079137 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070335 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:32:42.079137 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070337 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:32:42.079137 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070340 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:32:42.079734 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070343 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:32:42.079734 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070345 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:32:42.079734 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070349 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:32:42.079734 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070351 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:32:42.079734 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070354 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:32:42.079734 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070356 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:32:42.079734 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070359 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:32:42.079734 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070361 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:32:42.079734 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070364 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:32:42.079734 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070366 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:32:42.079734 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070369 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:32:42.079734 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070372 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:32:42.079734 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070374 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:32:42.079734 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070376 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:32:42.079734 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070379 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:32:42.079734 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070381 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:32:42.079734 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070384 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:32:42.079734 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070386 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:32:42.079734 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070389 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:32:42.079734 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070391 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:32:42.080442 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070394 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:32:42.080442 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070397 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:32:42.080442 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070400 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:32:42.080442 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070402 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:32:42.080442 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070405 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:32:42.080442 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070407 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:32:42.080442 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070410 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:32:42.080442 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070415 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:32:42.080442 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070419 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:32:42.080442 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070423 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:32:42.080442 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070427 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:32:42.080442 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070430 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:32:42.080442 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070432 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:32:42.080442 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070435 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:32:42.080442 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070438 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:32:42.080442 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070441 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:32:42.080442 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070444 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:32:42.080442 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070447 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:32:42.080442 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070449 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:32:42.080919 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070452 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:32:42.080919 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070455 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:32:42.080919 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070457 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:32:42.080919 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070460 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:32:42.080919 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070462 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:32:42.080919 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070465 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:32:42.080919 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070468 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:32:42.080919 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070470 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:32:42.080919 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070473 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:32:42.080919 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070476 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:32:42.080919 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070478 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:32:42.080919 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070481 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:32:42.080919 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070484 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:32:42.080919 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.070506 2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 18:32:42.080919 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070537 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:32:42.080919 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070541 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:32:42.081335 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070543 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:32:42.081335 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070546 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:32:42.081335 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070549 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:32:42.081335 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070552 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:32:42.081335 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070557 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:32:42.081335 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070562 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:32:42.081335 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070566 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:32:42.081335 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070569 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:32:42.081335 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070572 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:32:42.081335 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070575 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:32:42.081335 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070577 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:32:42.081335 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070581 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:32:42.081335 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070584 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:32:42.081335 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070586 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:32:42.081335 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070589 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:32:42.081335 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070591 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:32:42.081335 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070594 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:32:42.081335 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070597 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:32:42.081798 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070599 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:32:42.081798 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070602 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:32:42.081798 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070605 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:32:42.081798 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070607 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:32:42.081798 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070610 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:32:42.081798 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070612 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:32:42.081798 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070615 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:32:42.081798 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070618 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:32:42.081798 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070620 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:32:42.081798 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070623 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:32:42.081798 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070625 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:32:42.081798 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070628 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:32:42.081798 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070630 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:32:42.081798 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070633 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:32:42.081798 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070636 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:32:42.081798 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070638 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:32:42.081798 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070641 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:32:42.081798 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070643 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:32:42.081798 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070647 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:32:42.081798 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070651 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:32:42.082478 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070653 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:32:42.082478 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070656 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:32:42.082478 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070659 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:32:42.082478 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070661 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:32:42.082478 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070664 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:32:42.082478 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070666 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:32:42.082478 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070669 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:32:42.082478 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070672 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:32:42.082478 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070674 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:32:42.082478 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070677 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:32:42.082478 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070679 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:32:42.082478 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070682 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:32:42.082478 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070684 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:32:42.082478 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070687 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:32:42.082478 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070690 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:32:42.082478 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070692 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:32:42.082478 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070695 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:32:42.082478 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070697 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:32:42.082478 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070700 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:32:42.082478 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070702 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:32:42.082986 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070705 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:32:42.082986 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070708 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:32:42.082986 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070710 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:32:42.082986 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070713 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:32:42.082986 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070715 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:32:42.082986 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070718 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:32:42.082986 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070721 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:32:42.082986 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070723 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:32:42.082986 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070726 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:32:42.082986 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070728 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:32:42.082986 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070731 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:32:42.082986 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070735 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:32:42.082986 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070737 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:32:42.082986 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070740 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:32:42.082986 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070743 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:32:42.082986 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070745 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:32:42.082986 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070748 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:32:42.082986 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070751 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:32:42.082986 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070753 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:32:42.083481 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070756 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:32:42.083481 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070759 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:32:42.083481 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070762 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:32:42.083481 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070764 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:32:42.083481 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070767 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:32:42.083481 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070770 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:32:42.083481 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070772 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:32:42.083481 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070775 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:32:42.083481 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070778 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:32:42.083481 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070780 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:32:42.083481 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070783 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:32:42.083481 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070785 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:32:42.083481 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070788 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:32:42.083481 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070790 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:32:42.083481 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070793 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:32:42.083481 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070797 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:32:42.083481 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070799 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:32:42.083481 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070802 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:32:42.083481 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070804 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:32:42.083481 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070807 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:32:42.083936 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070810 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:32:42.083936 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070812 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:32:42.083936 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070815 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:32:42.083936 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070817 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:32:42.083936 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070821 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:32:42.083936 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070824 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:32:42.083936 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:42.070827 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:32:42.083936 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.070875 2575 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 18:32:42.083936 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.073839 2575 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 18:32:42.083936 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.075216 2575 server.go:1014] "Starting client certificate rotation" Oct 30 18:32:42.083936 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.075315 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 18:32:42.083936 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.075360 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 18:32:42.083936 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.077295 2575 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 18:32:42.083936 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.083136 2575 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 18:32:42.090389 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.090373 2575 log.go:25] "Validated CRI v1 runtime API" Oct 30 18:32:42.093450 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.093437 2575 log.go:25] "Validated CRI v1 image API" Oct 30 18:32:42.094721 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.094696 2575 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 18:32:42.097468 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.097447 2575 fs.go:135] Filesystem UUIDs: map[5faaf9ec-896a-4a34-b4bc-b7d4e78ea6b0:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 e22d9d11-52dc-4fb5-a2f5-fcb44834329f:/dev/nvme0n1p4] Oct 30 18:32:42.097529 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.097467 2575 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 18:32:42.103006 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.102894 2575 manager.go:217] Machine: {Timestamp:2026-10-30 18:32:42.101047795 +0000 UTC m=+0.320039209 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100031 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec26ec8f627b0e8f257c8877c68b21eb SystemUUID:ec26ec8f-627b-0e8f-257c-8877c68b21eb BootID:3b212ecd-8fbd-45c5-8b6e-b3b18acb359d Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:df:c6:5f:8b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:df:c6:5f:8b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:3e:ab:05:71:c4:54 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 18:32:42.103006 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.103005 2575 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 18:32:42.103180 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.103115 2575 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 18:32:42.104051 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.104025 2575 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 18:32:42.104205 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.104052 2575 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-163.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 18:32:42.104246 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.104208 2575 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 18:32:42.104246 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.104215 2575 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 18:32:42.104305 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.104258 2575 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 18:32:42.104305 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.104276 2575 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 18:32:42.104999 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.104989 2575 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 18:32:42.108594 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.108582 2575 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 18:32:42.108712 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.108702 2575 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 18:32:42.109353 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.109344 2575 kubelet.go:493] "Attempting to sync node with API server" Oct 30 18:32:42.109392 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.109357 2575 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 18:32:42.109392 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.109367 2575 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 18:32:42.109392 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.109376 2575 kubelet.go:405] "Adding apiserver pod source" Oct 30 18:32:42.109392 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.109384 2575 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 18:32:42.109835 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.109812 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 18:32:42.112240 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.112194 2575 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 18:32:42.116895 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.116871 2575 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 18:32:42.118105 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.118088 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 18:32:42.118823 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.118810 2575 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 18:32:42.118860 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.118848 2575 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 18:32:42.119004 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.118988 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 18:32:42.119004 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.119004 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 18:32:42.119110 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.119011 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 18:32:42.119110 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.119017 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 18:32:42.119110 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.119022 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 18:32:42.119110 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.119028 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 18:32:42.119582 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.119573 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 18:32:42.119582 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.119583 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 18:32:42.119642 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.119591 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 18:32:42.119642 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.119597 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 18:32:42.119642 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.119605 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 18:32:42.119642 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.119613 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 18:32:42.120856 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.120845 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 18:32:42.120856 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.120855 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 18:32:42.124507 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.124492 2575 server.go:1263] "Started kubelet" Oct 30 18:32:42.124616 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.124592 2575 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 18:32:42.124659 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.124585 2575 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 18:32:42.124688 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.124659 2575 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 18:32:42.125067 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.125049 2575 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-163.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 18:32:42.125310 ip-10-0-140-163 systemd[1]: Started Kubernetes Kubelet. Oct 30 18:32:42.125806 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.125725 2575 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 18:32:42.126878 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.126862 2575 server.go:317] "Adding debug handlers to kubelet server" Oct 30 18:32:42.128764 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:42.128738 2575 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 18:32:42.131163 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.131127 2575 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 18:32:42.131712 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.131138 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 18:32:42.131712 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:42.131272 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-163.us-east-2.compute.internal\" not found" Oct 30 18:32:42.131712 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.131675 2575 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 18:32:42.131712 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.131696 2575 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 18:32:42.131712 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.131706 2575 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 18:32:42.131980 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.131869 2575 reconstruct.go:97] "Volume reconstruction finished" Oct 30 18:32:42.131980 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.131879 2575 reconciler.go:29] "Reconciler: start to sync state" Oct 30 18:32:42.132066 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.132053 2575 factory.go:55] Registering systemd factory Oct 30 18:32:42.132104 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.132068 2575 factory.go:223] Registration of the systemd container factory successfully Oct 30 18:32:42.132302 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.132287 2575 factory.go:153] Registering CRI-O factory Oct 30 18:32:42.132366 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.132306 2575 factory.go:223] Registration of the crio container factory successfully Oct 30 18:32:42.132366 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.132352 2575 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 18:32:42.132433 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.132369 2575 factory.go:103] Registering Raw factory Oct 30 18:32:42.132433 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.132384 2575 manager.go:1199] Started watching for new ooms in manager Oct 30 18:32:42.133607 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.133591 2575 manager.go:319] Starting recovery of all containers Oct 30 18:32:42.135215 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:42.135180 2575 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-163.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 18:32:42.135549 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:42.135517 2575 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 18:32:42.135764 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:42.135719 2575 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-163.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 18:32:42.136795 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:42.135771 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-163.us-east-2.compute.internal.18e3624d9de2a6ac default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-163.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-163.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 18:32:42.124469932 +0000 UTC m=+0.343461345,LastTimestamp:2026-10-30 18:32:42.124469932 +0000 UTC m=+0.343461345,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-163.us-east-2.compute.internal,}" Oct 30 18:32:42.141595 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.141573 2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-lcbxb" Oct 30 18:32:42.141873 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:42.141849 2575 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 18:32:42.148555 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.148541 2575 manager.go:324] Recovery completed Oct 30 18:32:42.151766 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.151751 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-lcbxb" Oct 30 18:32:42.152481 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.152469 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 18:32:42.154435 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.154410 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-163.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 18:32:42.154500 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.154480 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-163.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 18:32:42.154532 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.154499 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-163.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 18:32:42.155073 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.155060 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 18:32:42.155165 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.155064 2575 cpu_manager.go:226] "Starting" policy="none" Oct 30 18:32:42.155165 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.155084 2575 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 18:32:42.155165 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.155099 2575 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 18:32:42.155760 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.155744 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-163.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 18:32:42.155828 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.155769 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-163.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 18:32:42.155828 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.155780 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-163.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 18:32:42.156102 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:42.156012 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-163.us-east-2.compute.internal.18e3624d9fac3cc0 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-163.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-163.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-163.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 18:32:42.154458304 +0000 UTC m=+0.373449721,LastTimestamp:2026-10-30 18:32:42.154458304 +0000 UTC m=+0.373449721,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-163.us-east-2.compute.internal,}" Oct 30 18:32:42.157192 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.157179 2575 policy_none.go:50] "Start" Oct 30 18:32:42.157192 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.157194 2575 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 18:32:42.157287 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.157204 2575 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 18:32:42.159292 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.159281 2575 policy_none.go:44] "Start" Oct 30 18:32:42.192845 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.192673 2575 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 18:32:42.198071 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.198056 2575 manager.go:341] "Starting Device Plugin manager" Oct 30 18:32:42.198184 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:42.198084 2575 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 18:32:42.198184 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.198092 2575 server.go:87] "Starting device plugin registration server" Oct 30 18:32:42.198340 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.198328 2575 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 18:32:42.198392 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.198340 2575 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 18:32:42.198457 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.198441 2575 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 18:32:42.198852 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.198566 2575 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 18:32:42.198852 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.198579 2575 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 18:32:42.199156 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:42.199074 2575 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 18:32:42.199156 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:42.199113 2575 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-163.us-east-2.compute.internal\" not found" Oct 30 18:32:42.298877 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.298850 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 18:32:42.300818 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.299809 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-163.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 18:32:42.300818 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.299838 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-163.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 18:32:42.300818 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.299849 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-163.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 18:32:42.300818 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.299876 2575 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-140-163.us-east-2.compute.internal" Oct 30 18:32:42.300818 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.300721 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 18:32:42.302060 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.302042 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 18:32:42.302105 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.302073 2575 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 18:32:42.302105 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.302096 2575 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 18:32:42.302191 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:42.302134 2575 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 18:32:42.307741 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.307695 2575 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 18:32:42.308244 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.308230 2575 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-140-163.us-east-2.compute.internal" Oct 30 18:32:42.308317 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:42.308251 2575 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-163.us-east-2.compute.internal\": node \"ip-10-0-140-163.us-east-2.compute.internal\" not found" Oct 30 18:32:42.344047 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:42.344027 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-163.us-east-2.compute.internal\" not found" Oct 30 18:32:42.402534 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.402508 2575 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-163.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-163.us-east-2.compute.internal"] Oct 30 18:32:42.402618 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.402591 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 18:32:42.403474 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.403461 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-163.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 18:32:42.403526 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.403490 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-163.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 18:32:42.403526 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.403504 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-163.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 18:32:42.405688 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.405675 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 18:32:42.406272 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.406250 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-163.us-east-2.compute.internal" Oct 30 18:32:42.406356 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.406292 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 18:32:42.406441 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.406428 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-163.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 18:32:42.406493 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.406453 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-163.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 18:32:42.406493 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.406463 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-163.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 18:32:42.407055 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.407041 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-163.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 18:32:42.407120 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.407068 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-163.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 18:32:42.407120 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.407078 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-163.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 18:32:42.408584 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.408565 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-163.us-east-2.compute.internal" Oct 30 18:32:42.408669 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.408591 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 18:32:42.409268 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.409253 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-163.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 18:32:42.409345 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.409278 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-163.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 18:32:42.409345 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.409288 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-163.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 18:32:42.429431 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:42.429414 2575 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-163.us-east-2.compute.internal\" not found" node="ip-10-0-140-163.us-east-2.compute.internal" Oct 30 18:32:42.431338 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:42.431323 2575 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-163.us-east-2.compute.internal\" not found" node="ip-10-0-140-163.us-east-2.compute.internal" Oct 30 18:32:42.434072 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.434043 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/40b5c76fae82f6d7a38a1a745d5d3947-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-163.us-east-2.compute.internal\" (UID: \"40b5c76fae82f6d7a38a1a745d5d3947\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-163.us-east-2.compute.internal" Oct 30 18:32:42.434120 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.434081 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3b404058373133d538844156a64ac398-config\") pod \"kube-apiserver-proxy-ip-10-0-140-163.us-east-2.compute.internal\" (UID: \"3b404058373133d538844156a64ac398\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-163.us-east-2.compute.internal" Oct 30 18:32:42.434120 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.434097 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/40b5c76fae82f6d7a38a1a745d5d3947-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-163.us-east-2.compute.internal\" (UID: \"40b5c76fae82f6d7a38a1a745d5d3947\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-163.us-east-2.compute.internal" Oct 30 18:32:42.444382 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:42.444360 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-163.us-east-2.compute.internal\" not found" Oct 30 18:32:42.534255 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.534215 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/40b5c76fae82f6d7a38a1a745d5d3947-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-163.us-east-2.compute.internal\" (UID: \"40b5c76fae82f6d7a38a1a745d5d3947\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-163.us-east-2.compute.internal" Oct 30 18:32:42.534255 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.534249 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/40b5c76fae82f6d7a38a1a745d5d3947-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-163.us-east-2.compute.internal\" (UID: \"40b5c76fae82f6d7a38a1a745d5d3947\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-163.us-east-2.compute.internal" Oct 30 18:32:42.534436 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.534266 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3b404058373133d538844156a64ac398-config\") pod \"kube-apiserver-proxy-ip-10-0-140-163.us-east-2.compute.internal\" (UID: \"3b404058373133d538844156a64ac398\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-163.us-east-2.compute.internal" Oct 30 18:32:42.534436 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.534293 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3b404058373133d538844156a64ac398-config\") pod \"kube-apiserver-proxy-ip-10-0-140-163.us-east-2.compute.internal\" (UID: \"3b404058373133d538844156a64ac398\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-163.us-east-2.compute.internal" Oct 30 18:32:42.534436 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.534304 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/40b5c76fae82f6d7a38a1a745d5d3947-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-163.us-east-2.compute.internal\" (UID: \"40b5c76fae82f6d7a38a1a745d5d3947\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-163.us-east-2.compute.internal" Oct 30 18:32:42.534436 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.534304 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/40b5c76fae82f6d7a38a1a745d5d3947-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-163.us-east-2.compute.internal\" (UID: \"40b5c76fae82f6d7a38a1a745d5d3947\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-163.us-east-2.compute.internal" Oct 30 18:32:42.544481 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:42.544461 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-163.us-east-2.compute.internal\" not found" Oct 30 18:32:42.644719 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:42.644657 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-163.us-east-2.compute.internal\" not found" Oct 30 18:32:42.731898 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.731870 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-163.us-east-2.compute.internal" Oct 30 18:32:42.735061 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:42.735045 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-163.us-east-2.compute.internal" Oct 30 18:32:42.745588 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:42.745571 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-163.us-east-2.compute.internal\" not found" Oct 30 18:32:42.846068 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:42.846039 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-163.us-east-2.compute.internal\" not found" Oct 30 18:32:42.946587 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:42.946518 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-163.us-east-2.compute.internal\" not found" Oct 30 18:32:43.046314 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.046287 2575 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 18:32:43.074536 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.074514 2575 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 18:32:43.074967 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.074627 2575 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 18:32:43.074967 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.074662 2575 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 18:32:43.109833 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.109801 2575 apiserver.go:52] "Watching apiserver" Oct 30 18:32:43.131420 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.131397 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 18:32:43.131802 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.131787 2575 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-163.us-east-2.compute.internal" Oct 30 18:32:43.134663 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.134647 2575 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 18:32:43.136626 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.135246 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-zmdjg","openshift-cluster-node-tuning-operator/tuned-rpzvk","openshift-dns/node-resolver-bk5br","openshift-image-registry/node-ca-qrdc6","openshift-multus/multus-additional-cni-plugins-nl4m8","openshift-network-diagnostics/network-check-target-26f57","openshift-network-operator/iptables-alerter-b876b","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtz4g","openshift-multus/multus-68r7k","openshift-multus/network-metrics-daemon-dtppz","kube-system/konnectivity-agent-4g8sn","openshift-ovn-kubernetes/ovnkube-node-czs4j"] Oct 30 18:32:43.137718 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.137702 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmdjg" Oct 30 18:32:43.137787 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:43.137770 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmdjg" podUID="e163025a-b19f-44e2-9058-b65f31bfbcc4" Oct 30 18:32:43.139695 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.139676 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rpzvk" Oct 30 18:32:43.141607 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.141590 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bk5br" Oct 30 18:32:43.143537 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.143523 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qrdc6" Oct 30 18:32:43.144127 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.144107 2575 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 18:32:43.144224 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.144190 2575 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-163.us-east-2.compute.internal" Oct 30 18:32:43.145574 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.145557 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-nl4m8" Oct 30 18:32:43.147501 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.147488 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26f57" Oct 30 18:32:43.147562 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:43.147535 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26f57" podUID="fb66d280-cf24-4549-8fdb-89f16b6080c1" Oct 30 18:32:43.149483 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.149469 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-b876b" Oct 30 18:32:43.151671 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.151655 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtz4g" Oct 30 18:32:43.153559 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.153540 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 18:32:43.153631 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.153564 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 18:32:43.153699 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.153680 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-59qb5\"" Oct 30 18:32:43.153860 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.153845 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-68r7k" Oct 30 18:32:43.156050 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.156007 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 18:27:43 +0000 UTC" deadline="2028-06-06 01:04:14.435008683 +0000 UTC" Oct 30 18:32:43.156167 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.156053 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14022h31m31.278959091s" Oct 30 18:32:43.156167 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.156034 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dtppz" Oct 30 18:32:43.156167 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:43.156103 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dtppz" podUID="90488e60-cb70-4afa-a54b-6f333d345e84" Oct 30 18:32:43.156467 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.156446 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 18:32:43.156467 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.156462 2575 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 18:32:43.156624 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.156469 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 18:32:43.156624 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.156543 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 18:32:43.156624 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.156597 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 18:32:43.156624 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.156611 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 18:32:43.156771 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.156630 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 18:32:43.156771 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.156636 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 18:32:43.156771 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.156646 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 18:32:43.158353 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.158201 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 18:32:43.158353 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.158230 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-jhsz4\"" Oct 30 18:32:43.158353 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.158236 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-lc92h\"" Oct 30 18:32:43.158353 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.158337 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-s26z4\"" Oct 30 18:32:43.158550 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.158389 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4g8sn" Oct 30 18:32:43.160408 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.160392 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-163.us-east-2.compute.internal"] Oct 30 18:32:43.160466 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.160415 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-140-163.us-east-2.compute.internal"] Oct 30 18:32:43.160505 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.160484 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:32:43.176041 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.176023 2575 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 18:32:43.176114 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.176079 2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-2f4mk" Oct 30 18:32:43.189327 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.189312 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-2f4mk" Oct 30 18:32:43.232480 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.232459 2575 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 18:32:43.237453 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.237426 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5b826abe-3f79-43ab-8813-2406d5c4dae9-cni-binary-copy\") pod \"multus-additional-cni-plugins-nl4m8\" (UID: \"5b826abe-3f79-43ab-8813-2406d5c4dae9\") " pod="openshift-multus/multus-additional-cni-plugins-nl4m8" Oct 30 18:32:43.237561 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.237465 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b0f929fc-a9ea-4935-9fe7-770c5e7d4de5-serviceca\") pod \"node-ca-qrdc6\" (UID: \"b0f929fc-a9ea-4935-9fe7-770c5e7d4de5\") " pod="openshift-image-registry/node-ca-qrdc6" Oct 30 18:32:43.237561 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.237530 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90488e60-cb70-4afa-a54b-6f333d345e84-metrics-certs\") pod \"network-metrics-daemon-dtppz\" (UID: \"90488e60-cb70-4afa-a54b-6f333d345e84\") " pod="openshift-multus/network-metrics-daemon-dtppz" Oct 30 18:32:43.237561 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.237556 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/eef45986-9475-4e3e-a3f9-fdb682474ff3-etc-sysconfig\") pod \"tuned-rpzvk\" (UID: \"eef45986-9475-4e3e-a3f9-fdb682474ff3\") " pod="openshift-cluster-node-tuning-operator/tuned-rpzvk" Oct 30 18:32:43.237705 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.237570 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eef45986-9475-4e3e-a3f9-fdb682474ff3-sys\") pod \"tuned-rpzvk\" (UID: \"eef45986-9475-4e3e-a3f9-fdb682474ff3\") " pod="openshift-cluster-node-tuning-operator/tuned-rpzvk" Oct 30 18:32:43.237705 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.237585 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b826abe-3f79-43ab-8813-2406d5c4dae9-system-cni-dir\") pod \"multus-additional-cni-plugins-nl4m8\" (UID: \"5b826abe-3f79-43ab-8813-2406d5c4dae9\") " pod="openshift-multus/multus-additional-cni-plugins-nl4m8" Oct 30 18:32:43.237705 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.237606 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e91ad8e0-05ce-420f-9594-445f4d48739c-socket-dir\") pod \"aws-ebs-csi-driver-node-qtz4g\" (UID: \"e91ad8e0-05ce-420f-9594-445f4d48739c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtz4g" Oct 30 18:32:43.237705 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.237640 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e91ad8e0-05ce-420f-9594-445f4d48739c-device-dir\") pod \"aws-ebs-csi-driver-node-qtz4g\" (UID: \"e91ad8e0-05ce-420f-9594-445f4d48739c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtz4g" Oct 30 18:32:43.237705 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.237666 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d2baf406-5501-4717-b5e1-d145edfe9ddc-host-run-netns\") pod \"multus-68r7k\" (UID: \"d2baf406-5501-4717-b5e1-d145edfe9ddc\") " pod="openshift-multus/multus-68r7k" Oct 30 18:32:43.237705 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.237687 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/81f52134-6a09-431f-9f80-412ae7bbead8-host-run-netns\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:32:43.237705 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.237702 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d2baf406-5501-4717-b5e1-d145edfe9ddc-cnibin\") pod \"multus-68r7k\" (UID: \"d2baf406-5501-4717-b5e1-d145edfe9ddc\") " pod="openshift-multus/multus-68r7k" Oct 30 18:32:43.238036 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.237729 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d2baf406-5501-4717-b5e1-d145edfe9ddc-multus-socket-dir-parent\") pod \"multus-68r7k\" (UID: \"d2baf406-5501-4717-b5e1-d145edfe9ddc\") " pod="openshift-multus/multus-68r7k" Oct 30 18:32:43.238036 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.237751 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b4961ad6-2fc3-4c01-a565-6f66068d2958-agent-certs\") pod \"konnectivity-agent-4g8sn\" (UID: \"b4961ad6-2fc3-4c01-a565-6f66068d2958\") " pod="kube-system/konnectivity-agent-4g8sn" Oct 30 18:32:43.238036 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.237789 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81f52134-6a09-431f-9f80-412ae7bbead8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:32:43.238036 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.237818 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/eef45986-9475-4e3e-a3f9-fdb682474ff3-etc-sysctl-conf\") pod \"tuned-rpzvk\" (UID: \"eef45986-9475-4e3e-a3f9-fdb682474ff3\") " pod="openshift-cluster-node-tuning-operator/tuned-rpzvk" Oct 30 18:32:43.238036 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.237854 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e91ad8e0-05ce-420f-9594-445f4d48739c-sys-fs\") pod \"aws-ebs-csi-driver-node-qtz4g\" (UID: \"e91ad8e0-05ce-420f-9594-445f4d48739c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtz4g" Oct 30 18:32:43.238036 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.237875 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d2baf406-5501-4717-b5e1-d145edfe9ddc-host-run-multus-certs\") pod \"multus-68r7k\" (UID: \"d2baf406-5501-4717-b5e1-d145edfe9ddc\") " pod="openshift-multus/multus-68r7k" Oct 30 18:32:43.238036 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.237890 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4s85\" (UniqueName: \"kubernetes.io/projected/d2baf406-5501-4717-b5e1-d145edfe9ddc-kube-api-access-m4s85\") pod \"multus-68r7k\" (UID: \"d2baf406-5501-4717-b5e1-d145edfe9ddc\") " pod="openshift-multus/multus-68r7k" Oct 30 18:32:43.238036 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.237933 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b4961ad6-2fc3-4c01-a565-6f66068d2958-konnectivity-ca\") pod \"konnectivity-agent-4g8sn\" (UID: \"b4961ad6-2fc3-4c01-a565-6f66068d2958\") " pod="kube-system/konnectivity-agent-4g8sn" Oct 30 18:32:43.238036 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.237956 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/05cad8b6-56f5-4bd1-b12d-b2ae456b7027-hosts-file\") pod \"node-resolver-bk5br\" (UID: \"05cad8b6-56f5-4bd1-b12d-b2ae456b7027\") " pod="openshift-dns/node-resolver-bk5br" Oct 30 18:32:43.238036 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.237971 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5b826abe-3f79-43ab-8813-2406d5c4dae9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nl4m8\" (UID: \"5b826abe-3f79-43ab-8813-2406d5c4dae9\") " pod="openshift-multus/multus-additional-cni-plugins-nl4m8" Oct 30 18:32:43.238036 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.238001 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5b826abe-3f79-43ab-8813-2406d5c4dae9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nl4m8\" (UID: \"5b826abe-3f79-43ab-8813-2406d5c4dae9\") " pod="openshift-multus/multus-additional-cni-plugins-nl4m8" Oct 30 18:32:43.238036 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.238030 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e91ad8e0-05ce-420f-9594-445f4d48739c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qtz4g\" (UID: \"e91ad8e0-05ce-420f-9594-445f4d48739c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtz4g" Oct 30 18:32:43.238678 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.238067 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d2baf406-5501-4717-b5e1-d145edfe9ddc-multus-cni-dir\") pod \"multus-68r7k\" (UID: \"d2baf406-5501-4717-b5e1-d145edfe9ddc\") " pod="openshift-multus/multus-68r7k" Oct 30 18:32:43.238678 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.238089 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d2baf406-5501-4717-b5e1-d145edfe9ddc-host-var-lib-cni-multus\") pod \"multus-68r7k\" (UID: \"d2baf406-5501-4717-b5e1-d145edfe9ddc\") " pod="openshift-multus/multus-68r7k" Oct 30 18:32:43.238678 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.238112 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d2baf406-5501-4717-b5e1-d145edfe9ddc-multus-conf-dir\") pod \"multus-68r7k\" (UID: \"d2baf406-5501-4717-b5e1-d145edfe9ddc\") " pod="openshift-multus/multus-68r7k" Oct 30 18:32:43.238678 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.238176 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/81f52134-6a09-431f-9f80-412ae7bbead8-run-systemd\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:32:43.238678 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.238214 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/81f52134-6a09-431f-9f80-412ae7bbead8-node-log\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:32:43.238678 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.238256 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05cad8b6-56f5-4bd1-b12d-b2ae456b7027-tmp-dir\") pod \"node-resolver-bk5br\" (UID: \"05cad8b6-56f5-4bd1-b12d-b2ae456b7027\") " pod="openshift-dns/node-resolver-bk5br" Oct 30 18:32:43.238678 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.238283 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/81f52134-6a09-431f-9f80-412ae7bbead8-systemd-units\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:32:43.238678 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.238321 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/81f52134-6a09-431f-9f80-412ae7bbead8-run-ovn\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:32:43.238678 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.238336 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/81f52134-6a09-431f-9f80-412ae7bbead8-log-socket\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:32:43.238678 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.238351 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdz5l\" (UniqueName: \"kubernetes.io/projected/81f52134-6a09-431f-9f80-412ae7bbead8-kube-api-access-rdz5l\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:32:43.238678 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.238379 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l927s\" (UniqueName: \"kubernetes.io/projected/ef86d9fa-a344-4886-b80c-89ce84f418ea-kube-api-access-l927s\") pod \"iptables-alerter-b876b\" (UID: \"ef86d9fa-a344-4886-b80c-89ce84f418ea\") " pod="openshift-network-operator/iptables-alerter-b876b" Oct 30 18:32:43.238678 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.238397 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/eef45986-9475-4e3e-a3f9-fdb682474ff3-run\") pod \"tuned-rpzvk\" (UID: \"eef45986-9475-4e3e-a3f9-fdb682474ff3\") " pod="openshift-cluster-node-tuning-operator/tuned-rpzvk" Oct 30 18:32:43.238678 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.238425 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eef45986-9475-4e3e-a3f9-fdb682474ff3-lib-modules\") pod \"tuned-rpzvk\" (UID: \"eef45986-9475-4e3e-a3f9-fdb682474ff3\") " pod="openshift-cluster-node-tuning-operator/tuned-rpzvk" Oct 30 18:32:43.238678 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.238454 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eef45986-9475-4e3e-a3f9-fdb682474ff3-var-lib-kubelet\") pod \"tuned-rpzvk\" (UID: \"eef45986-9475-4e3e-a3f9-fdb682474ff3\") " pod="openshift-cluster-node-tuning-operator/tuned-rpzvk" Oct 30 18:32:43.238678 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.238478 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/eef45986-9475-4e3e-a3f9-fdb682474ff3-etc-tuned\") pod \"tuned-rpzvk\" (UID: \"eef45986-9475-4e3e-a3f9-fdb682474ff3\") " pod="openshift-cluster-node-tuning-operator/tuned-rpzvk" Oct 30 18:32:43.238678 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.238510 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ef86d9fa-a344-4886-b80c-89ce84f418ea-iptables-alerter-script\") pod \"iptables-alerter-b876b\" (UID: \"ef86d9fa-a344-4886-b80c-89ce84f418ea\") " pod="openshift-network-operator/iptables-alerter-b876b" Oct 30 18:32:43.239513 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.238528 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb2bf\" (UniqueName: \"kubernetes.io/projected/e91ad8e0-05ce-420f-9594-445f4d48739c-kube-api-access-nb2bf\") pod \"aws-ebs-csi-driver-node-qtz4g\" (UID: \"e91ad8e0-05ce-420f-9594-445f4d48739c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtz4g" Oct 30 18:32:43.239513 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.238553 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vj8gn\" (UniqueName: \"kubernetes.io/projected/05cad8b6-56f5-4bd1-b12d-b2ae456b7027-kube-api-access-vj8gn\") pod \"node-resolver-bk5br\" (UID: \"05cad8b6-56f5-4bd1-b12d-b2ae456b7027\") " pod="openshift-dns/node-resolver-bk5br" Oct 30 18:32:43.239513 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.238584 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d2baf406-5501-4717-b5e1-d145edfe9ddc-host-var-lib-kubelet\") pod \"multus-68r7k\" (UID: \"d2baf406-5501-4717-b5e1-d145edfe9ddc\") " pod="openshift-multus/multus-68r7k" Oct 30 18:32:43.239513 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.238612 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d2baf406-5501-4717-b5e1-d145edfe9ddc-multus-daemon-config\") pod \"multus-68r7k\" (UID: \"d2baf406-5501-4717-b5e1-d145edfe9ddc\") " pod="openshift-multus/multus-68r7k" Oct 30 18:32:43.239513 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.238642 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eef45986-9475-4e3e-a3f9-fdb682474ff3-etc-kubernetes\") pod \"tuned-rpzvk\" (UID: \"eef45986-9475-4e3e-a3f9-fdb682474ff3\") " pod="openshift-cluster-node-tuning-operator/tuned-rpzvk" Oct 30 18:32:43.239513 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.238674 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eef45986-9475-4e3e-a3f9-fdb682474ff3-tmp\") pod \"tuned-rpzvk\" (UID: \"eef45986-9475-4e3e-a3f9-fdb682474ff3\") " pod="openshift-cluster-node-tuning-operator/tuned-rpzvk" Oct 30 18:32:43.239513 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.238695 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ef86d9fa-a344-4886-b80c-89ce84f418ea-host-slash\") pod \"iptables-alerter-b876b\" (UID: \"ef86d9fa-a344-4886-b80c-89ce84f418ea\") " pod="openshift-network-operator/iptables-alerter-b876b" Oct 30 18:32:43.239513 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.238759 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d2baf406-5501-4717-b5e1-d145edfe9ddc-host-run-k8s-cni-cncf-io\") pod \"multus-68r7k\" (UID: \"d2baf406-5501-4717-b5e1-d145edfe9ddc\") " pod="openshift-multus/multus-68r7k" Oct 30 18:32:43.239513 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.238800 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzfgl\" (UniqueName: \"kubernetes.io/projected/90488e60-cb70-4afa-a54b-6f333d345e84-kube-api-access-xzfgl\") pod \"network-metrics-daemon-dtppz\" (UID: \"90488e60-cb70-4afa-a54b-6f333d345e84\") " pod="openshift-multus/network-metrics-daemon-dtppz" Oct 30 18:32:43.239513 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.238821 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/eef45986-9475-4e3e-a3f9-fdb682474ff3-etc-sysctl-d\") pod \"tuned-rpzvk\" (UID: \"eef45986-9475-4e3e-a3f9-fdb682474ff3\") " pod="openshift-cluster-node-tuning-operator/tuned-rpzvk" Oct 30 18:32:43.239513 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.238855 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/81f52134-6a09-431f-9f80-412ae7bbead8-run-openvswitch\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:32:43.239513 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.238874 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/81f52134-6a09-431f-9f80-412ae7bbead8-host-cni-netd\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:32:43.239513 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.238888 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/81f52134-6a09-431f-9f80-412ae7bbead8-ovnkube-config\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:32:43.239513 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.238904 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/81f52134-6a09-431f-9f80-412ae7bbead8-env-overrides\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:32:43.239513 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.238929 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/eef45986-9475-4e3e-a3f9-fdb682474ff3-etc-modprobe-d\") pod \"tuned-rpzvk\" (UID: \"eef45986-9475-4e3e-a3f9-fdb682474ff3\") " pod="openshift-cluster-node-tuning-operator/tuned-rpzvk" Oct 30 18:32:43.239513 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.238947 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e91ad8e0-05ce-420f-9594-445f4d48739c-registration-dir\") pod \"aws-ebs-csi-driver-node-qtz4g\" (UID: \"e91ad8e0-05ce-420f-9594-445f4d48739c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtz4g" Oct 30 18:32:43.240230 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.238971 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d2baf406-5501-4717-b5e1-d145edfe9ddc-etc-kubernetes\") pod \"multus-68r7k\" (UID: \"d2baf406-5501-4717-b5e1-d145edfe9ddc\") " pod="openshift-multus/multus-68r7k" Oct 30 18:32:43.240230 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.238985 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e163025a-b19f-44e2-9058-b65f31bfbcc4-kubelet-config\") pod \"global-pull-secret-syncer-zmdjg\" (UID: \"e163025a-b19f-44e2-9058-b65f31bfbcc4\") " pod="kube-system/global-pull-secret-syncer-zmdjg" Oct 30 18:32:43.240230 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.239003 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rncj\" (UniqueName: \"kubernetes.io/projected/b0f929fc-a9ea-4935-9fe7-770c5e7d4de5-kube-api-access-4rncj\") pod \"node-ca-qrdc6\" (UID: \"b0f929fc-a9ea-4935-9fe7-770c5e7d4de5\") " pod="openshift-image-registry/node-ca-qrdc6" Oct 30 18:32:43.240230 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.239026 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/81f52134-6a09-431f-9f80-412ae7bbead8-var-lib-openvswitch\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:32:43.240230 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.239049 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/81f52134-6a09-431f-9f80-412ae7bbead8-etc-openvswitch\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:32:43.240230 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.239062 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eef45986-9475-4e3e-a3f9-fdb682474ff3-host\") pod \"tuned-rpzvk\" (UID: \"eef45986-9475-4e3e-a3f9-fdb682474ff3\") " pod="openshift-cluster-node-tuning-operator/tuned-rpzvk" Oct 30 18:32:43.240230 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.239089 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knxck\" (UniqueName: \"kubernetes.io/projected/eef45986-9475-4e3e-a3f9-fdb682474ff3-kube-api-access-knxck\") pod \"tuned-rpzvk\" (UID: \"eef45986-9475-4e3e-a3f9-fdb682474ff3\") " pod="openshift-cluster-node-tuning-operator/tuned-rpzvk" Oct 30 18:32:43.240230 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.239106 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxzzq\" (UniqueName: \"kubernetes.io/projected/5b826abe-3f79-43ab-8813-2406d5c4dae9-kube-api-access-kxzzq\") pod \"multus-additional-cni-plugins-nl4m8\" (UID: \"5b826abe-3f79-43ab-8813-2406d5c4dae9\") " pod="openshift-multus/multus-additional-cni-plugins-nl4m8" Oct 30 18:32:43.240230 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.239125 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e163025a-b19f-44e2-9058-b65f31bfbcc4-original-pull-secret\") pod \"global-pull-secret-syncer-zmdjg\" (UID: \"e163025a-b19f-44e2-9058-b65f31bfbcc4\") " pod="kube-system/global-pull-secret-syncer-zmdjg" Oct 30 18:32:43.240230 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.239162 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d2baf406-5501-4717-b5e1-d145edfe9ddc-cni-binary-copy\") pod \"multus-68r7k\" (UID: \"d2baf406-5501-4717-b5e1-d145edfe9ddc\") " pod="openshift-multus/multus-68r7k" Oct 30 18:32:43.240230 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.239207 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b0f929fc-a9ea-4935-9fe7-770c5e7d4de5-host\") pod \"node-ca-qrdc6\" (UID: \"b0f929fc-a9ea-4935-9fe7-770c5e7d4de5\") " pod="openshift-image-registry/node-ca-qrdc6" Oct 30 18:32:43.240230 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.239235 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d2baf406-5501-4717-b5e1-d145edfe9ddc-system-cni-dir\") pod \"multus-68r7k\" (UID: \"d2baf406-5501-4717-b5e1-d145edfe9ddc\") " pod="openshift-multus/multus-68r7k" Oct 30 18:32:43.240230 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.239277 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e163025a-b19f-44e2-9058-b65f31bfbcc4-dbus\") pod \"global-pull-secret-syncer-zmdjg\" (UID: \"e163025a-b19f-44e2-9058-b65f31bfbcc4\") " pod="kube-system/global-pull-secret-syncer-zmdjg" Oct 30 18:32:43.240230 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.239318 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/81f52134-6a09-431f-9f80-412ae7bbead8-ovn-node-metrics-cert\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:32:43.240230 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.239360 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/81f52134-6a09-431f-9f80-412ae7bbead8-ovnkube-script-lib\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:32:43.240230 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.239403 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5b826abe-3f79-43ab-8813-2406d5c4dae9-cnibin\") pod \"multus-additional-cni-plugins-nl4m8\" (UID: \"5b826abe-3f79-43ab-8813-2406d5c4dae9\") " pod="openshift-multus/multus-additional-cni-plugins-nl4m8" Oct 30 18:32:43.240914 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.239451 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e91ad8e0-05ce-420f-9594-445f4d48739c-etc-selinux\") pod \"aws-ebs-csi-driver-node-qtz4g\" (UID: \"e91ad8e0-05ce-420f-9594-445f4d48739c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtz4g" Oct 30 18:32:43.240914 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.239478 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d2baf406-5501-4717-b5e1-d145edfe9ddc-hostroot\") pod \"multus-68r7k\" (UID: \"d2baf406-5501-4717-b5e1-d145edfe9ddc\") " pod="openshift-multus/multus-68r7k" Oct 30 18:32:43.240914 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.239496 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztd9j\" (UniqueName: \"kubernetes.io/projected/fb66d280-cf24-4549-8fdb-89f16b6080c1-kube-api-access-ztd9j\") pod \"network-check-target-26f57\" (UID: \"fb66d280-cf24-4549-8fdb-89f16b6080c1\") " pod="openshift-network-diagnostics/network-check-target-26f57" Oct 30 18:32:43.240914 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.239543 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/81f52134-6a09-431f-9f80-412ae7bbead8-host-kubelet\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:32:43.240914 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.239586 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81f52134-6a09-431f-9f80-412ae7bbead8-host-run-ovn-kubernetes\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:32:43.240914 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.239627 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/81f52134-6a09-431f-9f80-412ae7bbead8-host-cni-bin\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:32:43.240914 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.239664 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/eef45986-9475-4e3e-a3f9-fdb682474ff3-etc-systemd\") pod \"tuned-rpzvk\" (UID: \"eef45986-9475-4e3e-a3f9-fdb682474ff3\") " pod="openshift-cluster-node-tuning-operator/tuned-rpzvk" Oct 30 18:32:43.240914 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.239724 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5b826abe-3f79-43ab-8813-2406d5c4dae9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nl4m8\" (UID: \"5b826abe-3f79-43ab-8813-2406d5c4dae9\") " pod="openshift-multus/multus-additional-cni-plugins-nl4m8" Oct 30 18:32:43.240914 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.239758 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d2baf406-5501-4717-b5e1-d145edfe9ddc-host-var-lib-cni-bin\") pod \"multus-68r7k\" (UID: \"d2baf406-5501-4717-b5e1-d145edfe9ddc\") " pod="openshift-multus/multus-68r7k" Oct 30 18:32:43.240914 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.239798 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/81f52134-6a09-431f-9f80-412ae7bbead8-host-slash\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:32:43.253440 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.253405 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 18:32:43.253552 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.253534 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 18:32:43.253552 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.253542 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 18:32:43.253646 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.253621 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 18:32:43.255054 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.255037 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 18:32:43.255054 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.255045 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 18:32:43.255054 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.255069 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 18:32:43.255054 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.255083 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-g44rk\"" Oct 30 18:32:43.255421 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.255200 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-82xdm\"" Oct 30 18:32:43.255421 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.255206 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-9fr66\"" Oct 30 18:32:43.255421 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.255318 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-qwdqr\"" Oct 30 18:32:43.255421 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.255339 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 18:32:43.256972 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.256854 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 18:32:43.256972 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.256869 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 18:32:43.256972 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.256891 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 18:32:43.256972 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.256898 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 18:32:43.256972 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.256918 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 18:32:43.256972 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.256935 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 18:32:43.256972 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.256941 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 18:32:43.256972 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.256951 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-fshmx\"" Oct 30 18:32:43.257407 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.257061 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 18:32:43.340906 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.340880 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d2baf406-5501-4717-b5e1-d145edfe9ddc-system-cni-dir\") pod \"multus-68r7k\" (UID: \"d2baf406-5501-4717-b5e1-d145edfe9ddc\") " pod="openshift-multus/multus-68r7k" Oct 30 18:32:43.340906 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.340907 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e163025a-b19f-44e2-9058-b65f31bfbcc4-dbus\") pod \"global-pull-secret-syncer-zmdjg\" (UID: \"e163025a-b19f-44e2-9058-b65f31bfbcc4\") " pod="kube-system/global-pull-secret-syncer-zmdjg" Oct 30 18:32:43.341210 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.340924 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/81f52134-6a09-431f-9f80-412ae7bbead8-ovn-node-metrics-cert\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:32:43.341210 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.340946 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/81f52134-6a09-431f-9f80-412ae7bbead8-ovnkube-script-lib\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:32:43.341210 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.340973 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5b826abe-3f79-43ab-8813-2406d5c4dae9-cnibin\") pod \"multus-additional-cni-plugins-nl4m8\" (UID: \"5b826abe-3f79-43ab-8813-2406d5c4dae9\") " pod="openshift-multus/multus-additional-cni-plugins-nl4m8" Oct 30 18:32:43.341210 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.341012 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d2baf406-5501-4717-b5e1-d145edfe9ddc-system-cni-dir\") pod \"multus-68r7k\" (UID: \"d2baf406-5501-4717-b5e1-d145edfe9ddc\") " pod="openshift-multus/multus-68r7k" Oct 30 18:32:43.341210 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.341000 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e91ad8e0-05ce-420f-9594-445f4d48739c-etc-selinux\") pod \"aws-ebs-csi-driver-node-qtz4g\" (UID: \"e91ad8e0-05ce-420f-9594-445f4d48739c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtz4g" Oct 30 18:32:43.341210 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.341047 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d2baf406-5501-4717-b5e1-d145edfe9ddc-hostroot\") pod \"multus-68r7k\" (UID: \"d2baf406-5501-4717-b5e1-d145edfe9ddc\") " pod="openshift-multus/multus-68r7k" Oct 30 18:32:43.341210 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.341064 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5b826abe-3f79-43ab-8813-2406d5c4dae9-cnibin\") pod \"multus-additional-cni-plugins-nl4m8\" (UID: \"5b826abe-3f79-43ab-8813-2406d5c4dae9\") " pod="openshift-multus/multus-additional-cni-plugins-nl4m8" Oct 30 18:32:43.341210 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.341070 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztd9j\" (UniqueName: \"kubernetes.io/projected/fb66d280-cf24-4549-8fdb-89f16b6080c1-kube-api-access-ztd9j\") pod \"network-check-target-26f57\" (UID: \"fb66d280-cf24-4549-8fdb-89f16b6080c1\") " pod="openshift-network-diagnostics/network-check-target-26f57" Oct 30 18:32:43.341210 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.341094 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/81f52134-6a09-431f-9f80-412ae7bbead8-host-kubelet\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:32:43.341210 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.341101 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d2baf406-5501-4717-b5e1-d145edfe9ddc-hostroot\") pod \"multus-68r7k\" (UID: \"d2baf406-5501-4717-b5e1-d145edfe9ddc\") " pod="openshift-multus/multus-68r7k" Oct 30 18:32:43.341210 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.341117 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81f52134-6a09-431f-9f80-412ae7bbead8-host-run-ovn-kubernetes\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:32:43.341210 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.341127 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e163025a-b19f-44e2-9058-b65f31bfbcc4-dbus\") pod \"global-pull-secret-syncer-zmdjg\" (UID: \"e163025a-b19f-44e2-9058-b65f31bfbcc4\") " pod="kube-system/global-pull-secret-syncer-zmdjg" Oct 30 18:32:43.341210 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.341156 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/81f52134-6a09-431f-9f80-412ae7bbead8-host-cni-bin\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:32:43.341210 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.341186 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/eef45986-9475-4e3e-a3f9-fdb682474ff3-etc-systemd\") pod \"tuned-rpzvk\" (UID: \"eef45986-9475-4e3e-a3f9-fdb682474ff3\") " pod="openshift-cluster-node-tuning-operator/tuned-rpzvk" Oct 30 18:32:43.341210 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.341207 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81f52134-6a09-431f-9f80-412ae7bbead8-host-run-ovn-kubernetes\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:32:43.341210 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.341216 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5b826abe-3f79-43ab-8813-2406d5c4dae9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nl4m8\" (UID: \"5b826abe-3f79-43ab-8813-2406d5c4dae9\") " pod="openshift-multus/multus-additional-cni-plugins-nl4m8" Oct 30 18:32:43.341786 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.341234 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/81f52134-6a09-431f-9f80-412ae7bbead8-host-kubelet\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:32:43.341786 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.341243 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d2baf406-5501-4717-b5e1-d145edfe9ddc-host-var-lib-cni-bin\") pod \"multus-68r7k\" (UID: \"d2baf406-5501-4717-b5e1-d145edfe9ddc\") " pod="openshift-multus/multus-68r7k" Oct 30 18:32:43.341786 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.341257 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/81f52134-6a09-431f-9f80-412ae7bbead8-host-cni-bin\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:32:43.341786 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.341266 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/81f52134-6a09-431f-9f80-412ae7bbead8-host-slash\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:32:43.341786 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.341291 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5b826abe-3f79-43ab-8813-2406d5c4dae9-cni-binary-copy\") pod \"multus-additional-cni-plugins-nl4m8\" (UID: \"5b826abe-3f79-43ab-8813-2406d5c4dae9\") " pod="openshift-multus/multus-additional-cni-plugins-nl4m8" Oct 30 18:32:43.341786 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.341291 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/eef45986-9475-4e3e-a3f9-fdb682474ff3-etc-systemd\") pod \"tuned-rpzvk\" (UID: \"eef45986-9475-4e3e-a3f9-fdb682474ff3\") " pod="openshift-cluster-node-tuning-operator/tuned-rpzvk" Oct 30 18:32:43.341786 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.341329 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b0f929fc-a9ea-4935-9fe7-770c5e7d4de5-serviceca\") pod \"node-ca-qrdc6\" (UID: \"b0f929fc-a9ea-4935-9fe7-770c5e7d4de5\") " pod="openshift-image-registry/node-ca-qrdc6" Oct 30 18:32:43.341786 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.341364 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90488e60-cb70-4afa-a54b-6f333d345e84-metrics-certs\") pod \"network-metrics-daemon-dtppz\" (UID: \"90488e60-cb70-4afa-a54b-6f333d345e84\") " pod="openshift-multus/network-metrics-daemon-dtppz" Oct 30 18:32:43.341786 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.341376 2575 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 18:32:43.341786 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.341388 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d2baf406-5501-4717-b5e1-d145edfe9ddc-host-var-lib-cni-bin\") pod \"multus-68r7k\" (UID: \"d2baf406-5501-4717-b5e1-d145edfe9ddc\") " pod="openshift-multus/multus-68r7k" Oct 30 18:32:43.341786 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.341391 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/eef45986-9475-4e3e-a3f9-fdb682474ff3-etc-sysconfig\") pod \"tuned-rpzvk\" (UID: \"eef45986-9475-4e3e-a3f9-fdb682474ff3\") " pod="openshift-cluster-node-tuning-operator/tuned-rpzvk" Oct 30 18:32:43.341786 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.341424 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eef45986-9475-4e3e-a3f9-fdb682474ff3-sys\") pod \"tuned-rpzvk\" (UID: \"eef45986-9475-4e3e-a3f9-fdb682474ff3\") " pod="openshift-cluster-node-tuning-operator/tuned-rpzvk" Oct 30 18:32:43.341786 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.341441 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/eef45986-9475-4e3e-a3f9-fdb682474ff3-etc-sysconfig\") pod \"tuned-rpzvk\" (UID: \"eef45986-9475-4e3e-a3f9-fdb682474ff3\") " pod="openshift-cluster-node-tuning-operator/tuned-rpzvk" Oct 30 18:32:43.341786 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.341448 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b826abe-3f79-43ab-8813-2406d5c4dae9-system-cni-dir\") pod \"multus-additional-cni-plugins-nl4m8\" (UID: \"5b826abe-3f79-43ab-8813-2406d5c4dae9\") " pod="openshift-multus/multus-additional-cni-plugins-nl4m8" Oct 30 18:32:43.341786 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.341484 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b826abe-3f79-43ab-8813-2406d5c4dae9-system-cni-dir\") pod \"multus-additional-cni-plugins-nl4m8\" (UID: \"5b826abe-3f79-43ab-8813-2406d5c4dae9\") " pod="openshift-multus/multus-additional-cni-plugins-nl4m8" Oct 30 18:32:43.341786 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.341528 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eef45986-9475-4e3e-a3f9-fdb682474ff3-sys\") pod \"tuned-rpzvk\" (UID: \"eef45986-9475-4e3e-a3f9-fdb682474ff3\") " pod="openshift-cluster-node-tuning-operator/tuned-rpzvk" Oct 30 18:32:43.341786 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.341566 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/81f52134-6a09-431f-9f80-412ae7bbead8-host-slash\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:32:43.341786 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:43.341643 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 18:32:43.342589 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:43.341701 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90488e60-cb70-4afa-a54b-6f333d345e84-metrics-certs podName:90488e60-cb70-4afa-a54b-6f333d345e84 nodeName:}" failed. No retries permitted until 2026-10-30 18:32:43.841682976 +0000 UTC m=+2.060674395 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/90488e60-cb70-4afa-a54b-6f333d345e84-metrics-certs") pod "network-metrics-daemon-dtppz" (UID: "90488e60-cb70-4afa-a54b-6f333d345e84") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 18:32:43.342589 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.341702 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/81f52134-6a09-431f-9f80-412ae7bbead8-ovnkube-script-lib\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:32:43.342589 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.341764 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5b826abe-3f79-43ab-8813-2406d5c4dae9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nl4m8\" (UID: \"5b826abe-3f79-43ab-8813-2406d5c4dae9\") " pod="openshift-multus/multus-additional-cni-plugins-nl4m8" Oct 30 18:32:43.342589 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.341801 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e91ad8e0-05ce-420f-9594-445f4d48739c-socket-dir\") pod \"aws-ebs-csi-driver-node-qtz4g\" (UID: \"e91ad8e0-05ce-420f-9594-445f4d48739c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtz4g" Oct 30 18:32:43.342589 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.341834 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e91ad8e0-05ce-420f-9594-445f4d48739c-device-dir\") pod \"aws-ebs-csi-driver-node-qtz4g\" (UID: \"e91ad8e0-05ce-420f-9594-445f4d48739c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtz4g" Oct 30 18:32:43.342589 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.341137 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e91ad8e0-05ce-420f-9594-445f4d48739c-etc-selinux\") pod \"aws-ebs-csi-driver-node-qtz4g\" (UID: \"e91ad8e0-05ce-420f-9594-445f4d48739c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtz4g" Oct 30 18:32:43.342589 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.341859 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d2baf406-5501-4717-b5e1-d145edfe9ddc-host-run-netns\") pod \"multus-68r7k\" (UID: \"d2baf406-5501-4717-b5e1-d145edfe9ddc\") " pod="openshift-multus/multus-68r7k" Oct 30 18:32:43.342589 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.341854 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b0f929fc-a9ea-4935-9fe7-770c5e7d4de5-serviceca\") pod \"node-ca-qrdc6\" (UID: \"b0f929fc-a9ea-4935-9fe7-770c5e7d4de5\") " pod="openshift-image-registry/node-ca-qrdc6" Oct 30 18:32:43.342589 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.341884 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/81f52134-6a09-431f-9f80-412ae7bbead8-host-run-netns\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:32:43.342589 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.341899 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e91ad8e0-05ce-420f-9594-445f4d48739c-socket-dir\") pod \"aws-ebs-csi-driver-node-qtz4g\" (UID: \"e91ad8e0-05ce-420f-9594-445f4d48739c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtz4g" Oct 30 18:32:43.342589 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.341916 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d2baf406-5501-4717-b5e1-d145edfe9ddc-host-run-netns\") pod \"multus-68r7k\" (UID: \"d2baf406-5501-4717-b5e1-d145edfe9ddc\") " pod="openshift-multus/multus-68r7k" Oct 30 18:32:43.342589 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.341860 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5b826abe-3f79-43ab-8813-2406d5c4dae9-cni-binary-copy\") pod \"multus-additional-cni-plugins-nl4m8\" (UID: \"5b826abe-3f79-43ab-8813-2406d5c4dae9\") " pod="openshift-multus/multus-additional-cni-plugins-nl4m8" Oct 30 18:32:43.342589 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.341906 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d2baf406-5501-4717-b5e1-d145edfe9ddc-cnibin\") pod \"multus-68r7k\" (UID: \"d2baf406-5501-4717-b5e1-d145edfe9ddc\") " pod="openshift-multus/multus-68r7k" Oct 30 18:32:43.342589 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.341955 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d2baf406-5501-4717-b5e1-d145edfe9ddc-cnibin\") pod \"multus-68r7k\" (UID: \"d2baf406-5501-4717-b5e1-d145edfe9ddc\") " pod="openshift-multus/multus-68r7k" Oct 30 18:32:43.342589 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.341975 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d2baf406-5501-4717-b5e1-d145edfe9ddc-multus-socket-dir-parent\") pod \"multus-68r7k\" (UID: \"d2baf406-5501-4717-b5e1-d145edfe9ddc\") " pod="openshift-multus/multus-68r7k" Oct 30 18:32:43.342589 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.341977 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/81f52134-6a09-431f-9f80-412ae7bbead8-host-run-netns\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:32:43.342589 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.341904 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e91ad8e0-05ce-420f-9594-445f4d48739c-device-dir\") pod \"aws-ebs-csi-driver-node-qtz4g\" (UID: \"e91ad8e0-05ce-420f-9594-445f4d48739c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtz4g" Oct 30 18:32:43.343503 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.342000 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b4961ad6-2fc3-4c01-a565-6f66068d2958-agent-certs\") pod \"konnectivity-agent-4g8sn\" (UID: \"b4961ad6-2fc3-4c01-a565-6f66068d2958\") " pod="kube-system/konnectivity-agent-4g8sn" Oct 30 18:32:43.343503 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.342015 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d2baf406-5501-4717-b5e1-d145edfe9ddc-multus-socket-dir-parent\") pod \"multus-68r7k\" (UID: \"d2baf406-5501-4717-b5e1-d145edfe9ddc\") " pod="openshift-multus/multus-68r7k" Oct 30 18:32:43.343503 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.342018 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81f52134-6a09-431f-9f80-412ae7bbead8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:32:43.343503 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.342040 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81f52134-6a09-431f-9f80-412ae7bbead8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:32:43.343503 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.342050 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/eef45986-9475-4e3e-a3f9-fdb682474ff3-etc-sysctl-conf\") pod \"tuned-rpzvk\" (UID: \"eef45986-9475-4e3e-a3f9-fdb682474ff3\") " pod="openshift-cluster-node-tuning-operator/tuned-rpzvk" Oct 30 18:32:43.343503 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.342074 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e91ad8e0-05ce-420f-9594-445f4d48739c-sys-fs\") pod \"aws-ebs-csi-driver-node-qtz4g\" (UID: \"e91ad8e0-05ce-420f-9594-445f4d48739c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtz4g" Oct 30 18:32:43.343503 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.342094 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d2baf406-5501-4717-b5e1-d145edfe9ddc-host-run-multus-certs\") pod \"multus-68r7k\" (UID: \"d2baf406-5501-4717-b5e1-d145edfe9ddc\") " pod="openshift-multus/multus-68r7k" Oct 30 18:32:43.343503 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.342128 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4s85\" (UniqueName: \"kubernetes.io/projected/d2baf406-5501-4717-b5e1-d145edfe9ddc-kube-api-access-m4s85\") pod \"multus-68r7k\" (UID: \"d2baf406-5501-4717-b5e1-d145edfe9ddc\") " pod="openshift-multus/multus-68r7k" Oct 30 18:32:43.343503 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.342135 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e91ad8e0-05ce-420f-9594-445f4d48739c-sys-fs\") pod \"aws-ebs-csi-driver-node-qtz4g\" (UID: \"e91ad8e0-05ce-420f-9594-445f4d48739c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtz4g" Oct 30 18:32:43.343503 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.342179 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b4961ad6-2fc3-4c01-a565-6f66068d2958-konnectivity-ca\") pod \"konnectivity-agent-4g8sn\" (UID: \"b4961ad6-2fc3-4c01-a565-6f66068d2958\") " pod="kube-system/konnectivity-agent-4g8sn" Oct 30 18:32:43.343503 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.342185 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/eef45986-9475-4e3e-a3f9-fdb682474ff3-etc-sysctl-conf\") pod \"tuned-rpzvk\" (UID: \"eef45986-9475-4e3e-a3f9-fdb682474ff3\") " pod="openshift-cluster-node-tuning-operator/tuned-rpzvk" Oct 30 18:32:43.343503 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.342208 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/05cad8b6-56f5-4bd1-b12d-b2ae456b7027-hosts-file\") pod \"node-resolver-bk5br\" (UID: \"05cad8b6-56f5-4bd1-b12d-b2ae456b7027\") " pod="openshift-dns/node-resolver-bk5br" Oct 30 18:32:43.343503 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.342208 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d2baf406-5501-4717-b5e1-d145edfe9ddc-host-run-multus-certs\") pod \"multus-68r7k\" (UID: \"d2baf406-5501-4717-b5e1-d145edfe9ddc\") " pod="openshift-multus/multus-68r7k" Oct 30 18:32:43.343503 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.342223 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5b826abe-3f79-43ab-8813-2406d5c4dae9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nl4m8\" (UID: \"5b826abe-3f79-43ab-8813-2406d5c4dae9\") " pod="openshift-multus/multus-additional-cni-plugins-nl4m8" Oct 30 18:32:43.343503 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.342283 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/05cad8b6-56f5-4bd1-b12d-b2ae456b7027-hosts-file\") pod \"node-resolver-bk5br\" (UID: \"05cad8b6-56f5-4bd1-b12d-b2ae456b7027\") " pod="openshift-dns/node-resolver-bk5br" Oct 30 18:32:43.343503 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.342321 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5b826abe-3f79-43ab-8813-2406d5c4dae9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nl4m8\" (UID: \"5b826abe-3f79-43ab-8813-2406d5c4dae9\") " pod="openshift-multus/multus-additional-cni-plugins-nl4m8" Oct 30 18:32:43.343503 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.342349 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e91ad8e0-05ce-420f-9594-445f4d48739c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qtz4g\" (UID: \"e91ad8e0-05ce-420f-9594-445f4d48739c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtz4g" Oct 30 18:32:43.344459 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.342372 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d2baf406-5501-4717-b5e1-d145edfe9ddc-multus-cni-dir\") pod \"multus-68r7k\" (UID: \"d2baf406-5501-4717-b5e1-d145edfe9ddc\") " pod="openshift-multus/multus-68r7k" Oct 30 18:32:43.344459 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.342396 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d2baf406-5501-4717-b5e1-d145edfe9ddc-host-var-lib-cni-multus\") pod \"multus-68r7k\" (UID: \"d2baf406-5501-4717-b5e1-d145edfe9ddc\") " pod="openshift-multus/multus-68r7k" Oct 30 18:32:43.344459 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.342417 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e91ad8e0-05ce-420f-9594-445f4d48739c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qtz4g\" (UID: \"e91ad8e0-05ce-420f-9594-445f4d48739c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtz4g" Oct 30 18:32:43.344459 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.342419 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d2baf406-5501-4717-b5e1-d145edfe9ddc-multus-conf-dir\") pod \"multus-68r7k\" (UID: \"d2baf406-5501-4717-b5e1-d145edfe9ddc\") " pod="openshift-multus/multus-68r7k" Oct 30 18:32:43.344459 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.342446 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d2baf406-5501-4717-b5e1-d145edfe9ddc-multus-cni-dir\") pod \"multus-68r7k\" (UID: \"d2baf406-5501-4717-b5e1-d145edfe9ddc\") " pod="openshift-multus/multus-68r7k" Oct 30 18:32:43.344459 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.342449 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d2baf406-5501-4717-b5e1-d145edfe9ddc-multus-conf-dir\") pod \"multus-68r7k\" (UID: \"d2baf406-5501-4717-b5e1-d145edfe9ddc\") " pod="openshift-multus/multus-68r7k" Oct 30 18:32:43.344459 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.342461 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/81f52134-6a09-431f-9f80-412ae7bbead8-run-systemd\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:32:43.344459 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.342480 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d2baf406-5501-4717-b5e1-d145edfe9ddc-host-var-lib-cni-multus\") pod \"multus-68r7k\" (UID: \"d2baf406-5501-4717-b5e1-d145edfe9ddc\") " pod="openshift-multus/multus-68r7k" Oct 30 18:32:43.344459 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.342487 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/81f52134-6a09-431f-9f80-412ae7bbead8-run-systemd\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:32:43.344459 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.342490 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/81f52134-6a09-431f-9f80-412ae7bbead8-node-log\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:32:43.344459 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.342514 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05cad8b6-56f5-4bd1-b12d-b2ae456b7027-tmp-dir\") pod \"node-resolver-bk5br\" (UID: \"05cad8b6-56f5-4bd1-b12d-b2ae456b7027\") " pod="openshift-dns/node-resolver-bk5br" Oct 30 18:32:43.344459 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.342530 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/81f52134-6a09-431f-9f80-412ae7bbead8-systemd-units\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:32:43.344459 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.342544 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/81f52134-6a09-431f-9f80-412ae7bbead8-run-ovn\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:32:43.344459 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.342588 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/81f52134-6a09-431f-9f80-412ae7bbead8-run-ovn\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:32:43.344459 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.342596 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/81f52134-6a09-431f-9f80-412ae7bbead8-systemd-units\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:32:43.344459 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.342627 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/81f52134-6a09-431f-9f80-412ae7bbead8-node-log\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:32:43.344459 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.342626 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/81f52134-6a09-431f-9f80-412ae7bbead8-log-socket\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:32:43.344459 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.342651 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/81f52134-6a09-431f-9f80-412ae7bbead8-log-socket\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:32:43.345350 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.342661 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdz5l\" (UniqueName: \"kubernetes.io/projected/81f52134-6a09-431f-9f80-412ae7bbead8-kube-api-access-rdz5l\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:32:43.345350 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.342684 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l927s\" (UniqueName: \"kubernetes.io/projected/ef86d9fa-a344-4886-b80c-89ce84f418ea-kube-api-access-l927s\") pod \"iptables-alerter-b876b\" (UID: \"ef86d9fa-a344-4886-b80c-89ce84f418ea\") " pod="openshift-network-operator/iptables-alerter-b876b" Oct 30 18:32:43.345350 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.342699 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/eef45986-9475-4e3e-a3f9-fdb682474ff3-run\") pod \"tuned-rpzvk\" (UID: \"eef45986-9475-4e3e-a3f9-fdb682474ff3\") " pod="openshift-cluster-node-tuning-operator/tuned-rpzvk" Oct 30 18:32:43.345350 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.342712 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eef45986-9475-4e3e-a3f9-fdb682474ff3-lib-modules\") pod \"tuned-rpzvk\" (UID: \"eef45986-9475-4e3e-a3f9-fdb682474ff3\") " pod="openshift-cluster-node-tuning-operator/tuned-rpzvk" Oct 30 18:32:43.345350 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.342733 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eef45986-9475-4e3e-a3f9-fdb682474ff3-var-lib-kubelet\") pod \"tuned-rpzvk\" (UID: \"eef45986-9475-4e3e-a3f9-fdb682474ff3\") " pod="openshift-cluster-node-tuning-operator/tuned-rpzvk" Oct 30 18:32:43.345350 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.342756 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/eef45986-9475-4e3e-a3f9-fdb682474ff3-etc-tuned\") pod \"tuned-rpzvk\" (UID: \"eef45986-9475-4e3e-a3f9-fdb682474ff3\") " pod="openshift-cluster-node-tuning-operator/tuned-rpzvk" Oct 30 18:32:43.345350 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.342758 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5b826abe-3f79-43ab-8813-2406d5c4dae9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nl4m8\" (UID: \"5b826abe-3f79-43ab-8813-2406d5c4dae9\") " pod="openshift-multus/multus-additional-cni-plugins-nl4m8" Oct 30 18:32:43.345350 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.342792 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ef86d9fa-a344-4886-b80c-89ce84f418ea-iptables-alerter-script\") pod \"iptables-alerter-b876b\" (UID: \"ef86d9fa-a344-4886-b80c-89ce84f418ea\") " pod="openshift-network-operator/iptables-alerter-b876b" Oct 30 18:32:43.345350 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.342821 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb2bf\" (UniqueName: \"kubernetes.io/projected/e91ad8e0-05ce-420f-9594-445f4d48739c-kube-api-access-nb2bf\") pod \"aws-ebs-csi-driver-node-qtz4g\" (UID: \"e91ad8e0-05ce-420f-9594-445f4d48739c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtz4g" Oct 30 18:32:43.345350 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.342845 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj8gn\" (UniqueName: \"kubernetes.io/projected/05cad8b6-56f5-4bd1-b12d-b2ae456b7027-kube-api-access-vj8gn\") pod \"node-resolver-bk5br\" (UID: \"05cad8b6-56f5-4bd1-b12d-b2ae456b7027\") " pod="openshift-dns/node-resolver-bk5br" Oct 30 18:32:43.345350 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.342869 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d2baf406-5501-4717-b5e1-d145edfe9ddc-host-var-lib-kubelet\") pod \"multus-68r7k\" (UID: \"d2baf406-5501-4717-b5e1-d145edfe9ddc\") " pod="openshift-multus/multus-68r7k" Oct 30 18:32:43.345350 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.342892 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d2baf406-5501-4717-b5e1-d145edfe9ddc-multus-daemon-config\") pod \"multus-68r7k\" (UID: \"d2baf406-5501-4717-b5e1-d145edfe9ddc\") " pod="openshift-multus/multus-68r7k" Oct 30 18:32:43.345350 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.342920 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eef45986-9475-4e3e-a3f9-fdb682474ff3-etc-kubernetes\") pod \"tuned-rpzvk\" (UID: \"eef45986-9475-4e3e-a3f9-fdb682474ff3\") " pod="openshift-cluster-node-tuning-operator/tuned-rpzvk" Oct 30 18:32:43.345350 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.342943 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eef45986-9475-4e3e-a3f9-fdb682474ff3-tmp\") pod \"tuned-rpzvk\" (UID: \"eef45986-9475-4e3e-a3f9-fdb682474ff3\") " pod="openshift-cluster-node-tuning-operator/tuned-rpzvk" Oct 30 18:32:43.345350 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.342967 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ef86d9fa-a344-4886-b80c-89ce84f418ea-host-slash\") pod \"iptables-alerter-b876b\" (UID: \"ef86d9fa-a344-4886-b80c-89ce84f418ea\") " pod="openshift-network-operator/iptables-alerter-b876b" Oct 30 18:32:43.345350 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.342994 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d2baf406-5501-4717-b5e1-d145edfe9ddc-host-run-k8s-cni-cncf-io\") pod \"multus-68r7k\" (UID: \"d2baf406-5501-4717-b5e1-d145edfe9ddc\") " pod="openshift-multus/multus-68r7k" Oct 30 18:32:43.345350 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.343019 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzfgl\" (UniqueName: \"kubernetes.io/projected/90488e60-cb70-4afa-a54b-6f333d345e84-kube-api-access-xzfgl\") pod \"network-metrics-daemon-dtppz\" (UID: \"90488e60-cb70-4afa-a54b-6f333d345e84\") " pod="openshift-multus/network-metrics-daemon-dtppz" Oct 30 18:32:43.346261 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.343045 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/eef45986-9475-4e3e-a3f9-fdb682474ff3-etc-sysctl-d\") pod \"tuned-rpzvk\" (UID: \"eef45986-9475-4e3e-a3f9-fdb682474ff3\") " pod="openshift-cluster-node-tuning-operator/tuned-rpzvk" Oct 30 18:32:43.346261 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.343070 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/81f52134-6a09-431f-9f80-412ae7bbead8-run-openvswitch\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:32:43.346261 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.343095 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/81f52134-6a09-431f-9f80-412ae7bbead8-host-cni-netd\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:32:43.346261 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.343116 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eef45986-9475-4e3e-a3f9-fdb682474ff3-etc-kubernetes\") pod \"tuned-rpzvk\" (UID: \"eef45986-9475-4e3e-a3f9-fdb682474ff3\") " pod="openshift-cluster-node-tuning-operator/tuned-rpzvk" Oct 30 18:32:43.346261 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.343118 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/81f52134-6a09-431f-9f80-412ae7bbead8-ovnkube-config\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:32:43.346261 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.343177 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5b826abe-3f79-43ab-8813-2406d5c4dae9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nl4m8\" (UID: \"5b826abe-3f79-43ab-8813-2406d5c4dae9\") " pod="openshift-multus/multus-additional-cni-plugins-nl4m8" Oct 30 18:32:43.346261 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.343195 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/81f52134-6a09-431f-9f80-412ae7bbead8-env-overrides\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:32:43.346261 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.343200 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/eef45986-9475-4e3e-a3f9-fdb682474ff3-run\") pod \"tuned-rpzvk\" (UID: \"eef45986-9475-4e3e-a3f9-fdb682474ff3\") " pod="openshift-cluster-node-tuning-operator/tuned-rpzvk" Oct 30 18:32:43.346261 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.343215 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b4961ad6-2fc3-4c01-a565-6f66068d2958-konnectivity-ca\") pod \"konnectivity-agent-4g8sn\" (UID: \"b4961ad6-2fc3-4c01-a565-6f66068d2958\") " pod="kube-system/konnectivity-agent-4g8sn" Oct 30 18:32:43.346261 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.343222 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/eef45986-9475-4e3e-a3f9-fdb682474ff3-etc-modprobe-d\") pod \"tuned-rpzvk\" (UID: \"eef45986-9475-4e3e-a3f9-fdb682474ff3\") " pod="openshift-cluster-node-tuning-operator/tuned-rpzvk" Oct 30 18:32:43.346261 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.343253 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e91ad8e0-05ce-420f-9594-445f4d48739c-registration-dir\") pod \"aws-ebs-csi-driver-node-qtz4g\" (UID: \"e91ad8e0-05ce-420f-9594-445f4d48739c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtz4g" Oct 30 18:32:43.346261 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.343276 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d2baf406-5501-4717-b5e1-d145edfe9ddc-etc-kubernetes\") pod \"multus-68r7k\" (UID: \"d2baf406-5501-4717-b5e1-d145edfe9ddc\") " pod="openshift-multus/multus-68r7k" Oct 30 18:32:43.346261 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.343293 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eef45986-9475-4e3e-a3f9-fdb682474ff3-var-lib-kubelet\") pod \"tuned-rpzvk\" (UID: \"eef45986-9475-4e3e-a3f9-fdb682474ff3\") " pod="openshift-cluster-node-tuning-operator/tuned-rpzvk" Oct 30 18:32:43.346261 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.343298 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e163025a-b19f-44e2-9058-b65f31bfbcc4-kubelet-config\") pod \"global-pull-secret-syncer-zmdjg\" (UID: \"e163025a-b19f-44e2-9058-b65f31bfbcc4\") " pod="kube-system/global-pull-secret-syncer-zmdjg" Oct 30 18:32:43.346261 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.343346 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e163025a-b19f-44e2-9058-b65f31bfbcc4-kubelet-config\") pod \"global-pull-secret-syncer-zmdjg\" (UID: \"e163025a-b19f-44e2-9058-b65f31bfbcc4\") " pod="kube-system/global-pull-secret-syncer-zmdjg" Oct 30 18:32:43.346261 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.343345 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rncj\" (UniqueName: \"kubernetes.io/projected/b0f929fc-a9ea-4935-9fe7-770c5e7d4de5-kube-api-access-4rncj\") pod \"node-ca-qrdc6\" (UID: \"b0f929fc-a9ea-4935-9fe7-770c5e7d4de5\") " pod="openshift-image-registry/node-ca-qrdc6" Oct 30 18:32:43.346261 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.343389 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/81f52134-6a09-431f-9f80-412ae7bbead8-var-lib-openvswitch\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:32:43.347091 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.343417 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/81f52134-6a09-431f-9f80-412ae7bbead8-etc-openvswitch\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:32:43.347091 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.343441 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eef45986-9475-4e3e-a3f9-fdb682474ff3-host\") pod \"tuned-rpzvk\" (UID: \"eef45986-9475-4e3e-a3f9-fdb682474ff3\") " pod="openshift-cluster-node-tuning-operator/tuned-rpzvk" Oct 30 18:32:43.347091 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.343470 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knxck\" (UniqueName: \"kubernetes.io/projected/eef45986-9475-4e3e-a3f9-fdb682474ff3-kube-api-access-knxck\") pod \"tuned-rpzvk\" (UID: \"eef45986-9475-4e3e-a3f9-fdb682474ff3\") " pod="openshift-cluster-node-tuning-operator/tuned-rpzvk" Oct 30 18:32:43.347091 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.343497 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxzzq\" (UniqueName: \"kubernetes.io/projected/5b826abe-3f79-43ab-8813-2406d5c4dae9-kube-api-access-kxzzq\") pod \"multus-additional-cni-plugins-nl4m8\" (UID: \"5b826abe-3f79-43ab-8813-2406d5c4dae9\") " pod="openshift-multus/multus-additional-cni-plugins-nl4m8" Oct 30 18:32:43.347091 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.343528 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e163025a-b19f-44e2-9058-b65f31bfbcc4-original-pull-secret\") pod \"global-pull-secret-syncer-zmdjg\" (UID: \"e163025a-b19f-44e2-9058-b65f31bfbcc4\") " pod="kube-system/global-pull-secret-syncer-zmdjg" Oct 30 18:32:43.347091 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.343553 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d2baf406-5501-4717-b5e1-d145edfe9ddc-cni-binary-copy\") pod \"multus-68r7k\" (UID: \"d2baf406-5501-4717-b5e1-d145edfe9ddc\") " pod="openshift-multus/multus-68r7k" Oct 30 18:32:43.347091 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.343575 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b0f929fc-a9ea-4935-9fe7-770c5e7d4de5-host\") pod \"node-ca-qrdc6\" (UID: \"b0f929fc-a9ea-4935-9fe7-770c5e7d4de5\") " pod="openshift-image-registry/node-ca-qrdc6" Oct 30 18:32:43.347091 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.343650 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b0f929fc-a9ea-4935-9fe7-770c5e7d4de5-host\") pod \"node-ca-qrdc6\" (UID: \"b0f929fc-a9ea-4935-9fe7-770c5e7d4de5\") " pod="openshift-image-registry/node-ca-qrdc6" Oct 30 18:32:43.347091 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.343653 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/81f52134-6a09-431f-9f80-412ae7bbead8-ovnkube-config\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:32:43.347091 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.343692 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d2baf406-5501-4717-b5e1-d145edfe9ddc-host-var-lib-kubelet\") pod \"multus-68r7k\" (UID: \"d2baf406-5501-4717-b5e1-d145edfe9ddc\") " pod="openshift-multus/multus-68r7k" Oct 30 18:32:43.347091 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.343770 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/81f52134-6a09-431f-9f80-412ae7bbead8-run-openvswitch\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:32:43.347091 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.343872 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/eef45986-9475-4e3e-a3f9-fdb682474ff3-etc-sysctl-d\") pod \"tuned-rpzvk\" (UID: \"eef45986-9475-4e3e-a3f9-fdb682474ff3\") " pod="openshift-cluster-node-tuning-operator/tuned-rpzvk" Oct 30 18:32:43.347091 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.343884 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ef86d9fa-a344-4886-b80c-89ce84f418ea-host-slash\") pod \"iptables-alerter-b876b\" (UID: \"ef86d9fa-a344-4886-b80c-89ce84f418ea\") " pod="openshift-network-operator/iptables-alerter-b876b" Oct 30 18:32:43.347091 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.343920 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d2baf406-5501-4717-b5e1-d145edfe9ddc-host-run-k8s-cni-cncf-io\") pod \"multus-68r7k\" (UID: \"d2baf406-5501-4717-b5e1-d145edfe9ddc\") " pod="openshift-multus/multus-68r7k" Oct 30 18:32:43.347091 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.343930 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/81f52134-6a09-431f-9f80-412ae7bbead8-host-cni-netd\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:32:43.347091 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.343957 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/81f52134-6a09-431f-9f80-412ae7bbead8-var-lib-openvswitch\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:32:43.347091 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.343138 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eef45986-9475-4e3e-a3f9-fdb682474ff3-lib-modules\") pod \"tuned-rpzvk\" (UID: \"eef45986-9475-4e3e-a3f9-fdb682474ff3\") " pod="openshift-cluster-node-tuning-operator/tuned-rpzvk" Oct 30 18:32:43.347091 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.344167 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/81f52134-6a09-431f-9f80-412ae7bbead8-etc-openvswitch\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:32:43.347967 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.344222 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eef45986-9475-4e3e-a3f9-fdb682474ff3-host\") pod \"tuned-rpzvk\" (UID: \"eef45986-9475-4e3e-a3f9-fdb682474ff3\") " pod="openshift-cluster-node-tuning-operator/tuned-rpzvk" Oct 30 18:32:43.347967 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:43.344532 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 18:32:43.347967 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:43.344587 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e163025a-b19f-44e2-9058-b65f31bfbcc4-original-pull-secret podName:e163025a-b19f-44e2-9058-b65f31bfbcc4 nodeName:}" failed. No retries permitted until 2026-10-30 18:32:43.844570589 +0000 UTC m=+2.063562026 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e163025a-b19f-44e2-9058-b65f31bfbcc4-original-pull-secret") pod "global-pull-secret-syncer-zmdjg" (UID: "e163025a-b19f-44e2-9058-b65f31bfbcc4") : object "kube-system"/"original-pull-secret" not registered Oct 30 18:32:43.347967 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.344641 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05cad8b6-56f5-4bd1-b12d-b2ae456b7027-tmp-dir\") pod \"node-resolver-bk5br\" (UID: \"05cad8b6-56f5-4bd1-b12d-b2ae456b7027\") " pod="openshift-dns/node-resolver-bk5br" Oct 30 18:32:43.347967 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.344742 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e91ad8e0-05ce-420f-9594-445f4d48739c-registration-dir\") pod \"aws-ebs-csi-driver-node-qtz4g\" (UID: \"e91ad8e0-05ce-420f-9594-445f4d48739c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtz4g" Oct 30 18:32:43.347967 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.344837 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/eef45986-9475-4e3e-a3f9-fdb682474ff3-etc-modprobe-d\") pod \"tuned-rpzvk\" (UID: \"eef45986-9475-4e3e-a3f9-fdb682474ff3\") " pod="openshift-cluster-node-tuning-operator/tuned-rpzvk" Oct 30 18:32:43.347967 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.344842 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d2baf406-5501-4717-b5e1-d145edfe9ddc-etc-kubernetes\") pod \"multus-68r7k\" (UID: \"d2baf406-5501-4717-b5e1-d145edfe9ddc\") " pod="openshift-multus/multus-68r7k" Oct 30 18:32:43.347967 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.344846 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d2baf406-5501-4717-b5e1-d145edfe9ddc-multus-daemon-config\") pod \"multus-68r7k\" (UID: \"d2baf406-5501-4717-b5e1-d145edfe9ddc\") " pod="openshift-multus/multus-68r7k" Oct 30 18:32:43.347967 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.345451 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ef86d9fa-a344-4886-b80c-89ce84f418ea-iptables-alerter-script\") pod \"iptables-alerter-b876b\" (UID: \"ef86d9fa-a344-4886-b80c-89ce84f418ea\") " pod="openshift-network-operator/iptables-alerter-b876b" Oct 30 18:32:43.347967 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.345574 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/81f52134-6a09-431f-9f80-412ae7bbead8-env-overrides\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:32:43.347967 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.345765 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b4961ad6-2fc3-4c01-a565-6f66068d2958-agent-certs\") pod \"konnectivity-agent-4g8sn\" (UID: \"b4961ad6-2fc3-4c01-a565-6f66068d2958\") " pod="kube-system/konnectivity-agent-4g8sn" Oct 30 18:32:43.347967 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.345791 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/81f52134-6a09-431f-9f80-412ae7bbead8-ovn-node-metrics-cert\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:32:43.347967 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.345918 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d2baf406-5501-4717-b5e1-d145edfe9ddc-cni-binary-copy\") pod \"multus-68r7k\" (UID: \"d2baf406-5501-4717-b5e1-d145edfe9ddc\") " pod="openshift-multus/multus-68r7k" Oct 30 18:32:43.347967 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.345957 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eef45986-9475-4e3e-a3f9-fdb682474ff3-tmp\") pod \"tuned-rpzvk\" (UID: \"eef45986-9475-4e3e-a3f9-fdb682474ff3\") " pod="openshift-cluster-node-tuning-operator/tuned-rpzvk" Oct 30 18:32:43.348647 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.348457 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/eef45986-9475-4e3e-a3f9-fdb682474ff3-etc-tuned\") pod \"tuned-rpzvk\" (UID: \"eef45986-9475-4e3e-a3f9-fdb682474ff3\") " pod="openshift-cluster-node-tuning-operator/tuned-rpzvk" Oct 30 18:32:43.357614 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.357595 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4s85\" (UniqueName: \"kubernetes.io/projected/d2baf406-5501-4717-b5e1-d145edfe9ddc-kube-api-access-m4s85\") pod \"multus-68r7k\" (UID: \"d2baf406-5501-4717-b5e1-d145edfe9ddc\") " pod="openshift-multus/multus-68r7k" Oct 30 18:32:43.358436 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:43.358422 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 18:32:43.358492 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:43.358439 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 18:32:43.358492 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:43.358449 2575 projected.go:196] Error preparing data for projected volume kube-api-access-ztd9j for pod openshift-network-diagnostics/network-check-target-26f57: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 18:32:43.358492 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:43.358489 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fb66d280-cf24-4549-8fdb-89f16b6080c1-kube-api-access-ztd9j podName:fb66d280-cf24-4549-8fdb-89f16b6080c1 nodeName:}" failed. No retries permitted until 2026-10-30 18:32:43.858477992 +0000 UTC m=+2.077469411 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-ztd9j" (UniqueName: "kubernetes.io/projected/fb66d280-cf24-4549-8fdb-89f16b6080c1-kube-api-access-ztd9j") pod "network-check-target-26f57" (UID: "fb66d280-cf24-4549-8fdb-89f16b6080c1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 18:32:43.364834 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.364805 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzfgl\" (UniqueName: \"kubernetes.io/projected/90488e60-cb70-4afa-a54b-6f333d345e84-kube-api-access-xzfgl\") pod \"network-metrics-daemon-dtppz\" (UID: \"90488e60-cb70-4afa-a54b-6f333d345e84\") " pod="openshift-multus/network-metrics-daemon-dtppz" Oct 30 18:32:43.365546 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.365528 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knxck\" (UniqueName: \"kubernetes.io/projected/eef45986-9475-4e3e-a3f9-fdb682474ff3-kube-api-access-knxck\") pod \"tuned-rpzvk\" (UID: \"eef45986-9475-4e3e-a3f9-fdb682474ff3\") " pod="openshift-cluster-node-tuning-operator/tuned-rpzvk" Oct 30 18:32:43.370208 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.370183 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj8gn\" (UniqueName: \"kubernetes.io/projected/05cad8b6-56f5-4bd1-b12d-b2ae456b7027-kube-api-access-vj8gn\") pod \"node-resolver-bk5br\" (UID: \"05cad8b6-56f5-4bd1-b12d-b2ae456b7027\") " pod="openshift-dns/node-resolver-bk5br" Oct 30 18:32:43.370449 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.370430 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxzzq\" (UniqueName: \"kubernetes.io/projected/5b826abe-3f79-43ab-8813-2406d5c4dae9-kube-api-access-kxzzq\") pod \"multus-additional-cni-plugins-nl4m8\" (UID: \"5b826abe-3f79-43ab-8813-2406d5c4dae9\") " pod="openshift-multus/multus-additional-cni-plugins-nl4m8" Oct 30 18:32:43.370722 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.370700 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rncj\" (UniqueName: \"kubernetes.io/projected/b0f929fc-a9ea-4935-9fe7-770c5e7d4de5-kube-api-access-4rncj\") pod \"node-ca-qrdc6\" (UID: \"b0f929fc-a9ea-4935-9fe7-770c5e7d4de5\") " pod="openshift-image-registry/node-ca-qrdc6" Oct 30 18:32:43.371448 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.371431 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l927s\" (UniqueName: \"kubernetes.io/projected/ef86d9fa-a344-4886-b80c-89ce84f418ea-kube-api-access-l927s\") pod \"iptables-alerter-b876b\" (UID: \"ef86d9fa-a344-4886-b80c-89ce84f418ea\") " pod="openshift-network-operator/iptables-alerter-b876b" Oct 30 18:32:43.371685 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.371665 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb2bf\" (UniqueName: \"kubernetes.io/projected/e91ad8e0-05ce-420f-9594-445f4d48739c-kube-api-access-nb2bf\") pod \"aws-ebs-csi-driver-node-qtz4g\" (UID: \"e91ad8e0-05ce-420f-9594-445f4d48739c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtz4g" Oct 30 18:32:43.372475 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.372456 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdz5l\" (UniqueName: \"kubernetes.io/projected/81f52134-6a09-431f-9f80-412ae7bbead8-kube-api-access-rdz5l\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:32:43.408227 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:43.408203 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40b5c76fae82f6d7a38a1a745d5d3947.slice/crio-c9ae698e2ddbafabe23051c119ab3b29049da90efcd4e436acdec3095cd3fa73 WatchSource:0}: Error finding container c9ae698e2ddbafabe23051c119ab3b29049da90efcd4e436acdec3095cd3fa73: Status 404 returned error can't find the container with id c9ae698e2ddbafabe23051c119ab3b29049da90efcd4e436acdec3095cd3fa73 Oct 30 18:32:43.413102 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.413089 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 18:32:43.417475 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:43.417457 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b404058373133d538844156a64ac398.slice/crio-172feea2873431f0fbd887a83aadcc45053c61f7631d004fbfbf5d3f0aa4dd77 WatchSource:0}: Error finding container 172feea2873431f0fbd887a83aadcc45053c61f7631d004fbfbf5d3f0aa4dd77: Status 404 returned error can't find the container with id 172feea2873431f0fbd887a83aadcc45053c61f7631d004fbfbf5d3f0aa4dd77 Oct 30 18:32:43.474750 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.474690 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rpzvk" Oct 30 18:32:43.483718 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:43.483694 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeef45986_9475_4e3e_a3f9_fdb682474ff3.slice/crio-1cd2fe140d1ce2267185d56ec36bf3e19fb7cfc7194597c310311fe7067f8959 WatchSource:0}: Error finding container 1cd2fe140d1ce2267185d56ec36bf3e19fb7cfc7194597c310311fe7067f8959: Status 404 returned error can't find the container with id 1cd2fe140d1ce2267185d56ec36bf3e19fb7cfc7194597c310311fe7067f8959 Oct 30 18:32:43.491227 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.491203 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bk5br" Oct 30 18:32:43.499370 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:43.499350 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05cad8b6_56f5_4bd1_b12d_b2ae456b7027.slice/crio-8ae7cf2fa0bd11e84801e93bc28861efbd30ad203a94848ef0f5199e314f4f4c WatchSource:0}: Error finding container 8ae7cf2fa0bd11e84801e93bc28861efbd30ad203a94848ef0f5199e314f4f4c: Status 404 returned error can't find the container with id 8ae7cf2fa0bd11e84801e93bc28861efbd30ad203a94848ef0f5199e314f4f4c Oct 30 18:32:43.504197 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.504179 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qrdc6" Oct 30 18:32:43.511773 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:43.511753 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0f929fc_a9ea_4935_9fe7_770c5e7d4de5.slice/crio-2df7bb0c79d3346dc1213c978fbd1bf6cf2feee082618ee2378c05df610217e3 WatchSource:0}: Error finding container 2df7bb0c79d3346dc1213c978fbd1bf6cf2feee082618ee2378c05df610217e3: Status 404 returned error can't find the container with id 2df7bb0c79d3346dc1213c978fbd1bf6cf2feee082618ee2378c05df610217e3 Oct 30 18:32:43.533929 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.533910 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-b876b" Oct 30 18:32:43.537674 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.537658 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-nl4m8" Oct 30 18:32:43.539850 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.539833 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtz4g" Oct 30 18:32:43.541877 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:43.541856 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef86d9fa_a344_4886_b80c_89ce84f418ea.slice/crio-a2a9935a2a2d2b39234f40a39264586ed51525af2ed3b73ad1b2f5be53e581a7 WatchSource:0}: Error finding container a2a9935a2a2d2b39234f40a39264586ed51525af2ed3b73ad1b2f5be53e581a7: Status 404 returned error can't find the container with id a2a9935a2a2d2b39234f40a39264586ed51525af2ed3b73ad1b2f5be53e581a7 Oct 30 18:32:43.547016 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:43.546994 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b826abe_3f79_43ab_8813_2406d5c4dae9.slice/crio-3fc6a6002c134d13eff65b44fca6ed477f75262fb065531496148ca5cc8559e7 WatchSource:0}: Error finding container 3fc6a6002c134d13eff65b44fca6ed477f75262fb065531496148ca5cc8559e7: Status 404 returned error can't find the container with id 3fc6a6002c134d13eff65b44fca6ed477f75262fb065531496148ca5cc8559e7 Oct 30 18:32:43.550275 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:43.550257 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode91ad8e0_05ce_420f_9594_445f4d48739c.slice/crio-908390dea71dfceecda52d389bc886f06dadc8f7acb975b9402fe99fc931d605 WatchSource:0}: Error finding container 908390dea71dfceecda52d389bc886f06dadc8f7acb975b9402fe99fc931d605: Status 404 returned error can't find the container with id 908390dea71dfceecda52d389bc886f06dadc8f7acb975b9402fe99fc931d605 Oct 30 18:32:43.557252 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.557237 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-68r7k" Oct 30 18:32:43.565808 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:43.565783 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2baf406_5501_4717_b5e1_d145edfe9ddc.slice/crio-f653dd2bb9642410489171060113e275c78337a3f25db513e81c1722e9966388 WatchSource:0}: Error finding container f653dd2bb9642410489171060113e275c78337a3f25db513e81c1722e9966388: Status 404 returned error can't find the container with id f653dd2bb9642410489171060113e275c78337a3f25db513e81c1722e9966388 Oct 30 18:32:43.569093 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.569069 2575 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 18:32:43.573922 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.573900 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4g8sn" Oct 30 18:32:43.583278 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:43.583254 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4961ad6_2fc3_4c01_a565_6f66068d2958.slice/crio-56cf64efd808a743e34aa2da5f607cdce99f15da9da3be52befbd6612366c7d4 WatchSource:0}: Error finding container 56cf64efd808a743e34aa2da5f607cdce99f15da9da3be52befbd6612366c7d4: Status 404 returned error can't find the container with id 56cf64efd808a743e34aa2da5f607cdce99f15da9da3be52befbd6612366c7d4 Oct 30 18:32:43.593415 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.593397 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:32:43.602408 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:32:43.602388 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81f52134_6a09_431f_9f80_412ae7bbead8.slice/crio-ebb7a6cb2f181cb9daf6993af1f5e8a539dc87cc81cd568fb2eb857afff8935c WatchSource:0}: Error finding container ebb7a6cb2f181cb9daf6993af1f5e8a539dc87cc81cd568fb2eb857afff8935c: Status 404 returned error can't find the container with id ebb7a6cb2f181cb9daf6993af1f5e8a539dc87cc81cd568fb2eb857afff8935c Oct 30 18:32:43.604229 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:43.604206 2575 kuberuntime_manager.go:1707] "Unhandled Error" err="container ovn-controller start failed in pod ovnkube-node-czs4j_openshift-ovn-kubernetes(81f52134-6a09-431f-9f80-412ae7bbead8): ErrImagePull: pull QPS exceeded" logger="UnhandledError" Oct 30 18:32:43.605994 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:43.605979 2575 kuberuntime_manager.go:1707] "Unhandled Error" err="container kube-rbac-proxy-node start failed in pod ovnkube-node-czs4j_openshift-ovn-kubernetes(81f52134-6a09-431f-9f80-412ae7bbead8): ErrImagePull: pull QPS exceeded" logger="UnhandledError" Oct 30 18:32:43.610362 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:43.610326 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"ovn-controller\" with ErrImagePull: \"pull QPS exceeded\", failed to \"StartContainer\" for \"ovn-acl-logging\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:be9d8303ae4f79715ade971ddecdf6044ee6bb7b8d8b3a2f2f1e3bfe4e78c15a\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"kube-rbac-proxy-node\" with ErrImagePull: \"pull QPS exceeded\", failed to \"StartContainer\" for \"kube-rbac-proxy-ovn-metrics\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d017b7d199b1d54e889bb1eb6e572d8e6d94ad92610894009ee0562c2a5c0f51\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"northd\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:be9d8303ae4f79715ade971ddecdf6044ee6bb7b8d8b3a2f2f1e3bfe4e78c15a\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"nbdb\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:be9d8303ae4f79715ade971ddecdf6044ee6bb7b8d8b3a2f2f1e3bfe4e78c15a\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"sbdb\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:be9d8303ae4f79715ade971ddecdf6044ee6bb7b8d8b3a2f2f1e3bfe4e78c15a\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"ovnkube-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:be9d8303ae4f79715ade971ddecdf6044ee6bb7b8d8b3a2f2f1e3bfe4e78c15a\\\": ErrImagePull: pull QPS exceeded\"]" pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" podUID="81f52134-6a09-431f-9f80-412ae7bbead8" Oct 30 18:32:43.716441 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.716414 2575 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 18:32:43.846896 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.846770 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90488e60-cb70-4afa-a54b-6f333d345e84-metrics-certs\") pod \"network-metrics-daemon-dtppz\" (UID: \"90488e60-cb70-4afa-a54b-6f333d345e84\") " pod="openshift-multus/network-metrics-daemon-dtppz" Oct 30 18:32:43.846896 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.846833 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e163025a-b19f-44e2-9058-b65f31bfbcc4-original-pull-secret\") pod \"global-pull-secret-syncer-zmdjg\" (UID: \"e163025a-b19f-44e2-9058-b65f31bfbcc4\") " pod="kube-system/global-pull-secret-syncer-zmdjg" Oct 30 18:32:43.847068 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:43.846931 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 18:32:43.847068 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:43.846943 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 18:32:43.847068 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:43.847002 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e163025a-b19f-44e2-9058-b65f31bfbcc4-original-pull-secret podName:e163025a-b19f-44e2-9058-b65f31bfbcc4 nodeName:}" failed. No retries permitted until 2026-10-30 18:32:44.846984513 +0000 UTC m=+3.065975929 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e163025a-b19f-44e2-9058-b65f31bfbcc4-original-pull-secret") pod "global-pull-secret-syncer-zmdjg" (UID: "e163025a-b19f-44e2-9058-b65f31bfbcc4") : object "kube-system"/"original-pull-secret" not registered Oct 30 18:32:43.847068 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:43.847021 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90488e60-cb70-4afa-a54b-6f333d345e84-metrics-certs podName:90488e60-cb70-4afa-a54b-6f333d345e84 nodeName:}" failed. No retries permitted until 2026-10-30 18:32:44.847011161 +0000 UTC m=+3.066002568 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/90488e60-cb70-4afa-a54b-6f333d345e84-metrics-certs") pod "network-metrics-daemon-dtppz" (UID: "90488e60-cb70-4afa-a54b-6f333d345e84") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 18:32:43.948089 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:43.948059 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztd9j\" (UniqueName: \"kubernetes.io/projected/fb66d280-cf24-4549-8fdb-89f16b6080c1-kube-api-access-ztd9j\") pod \"network-check-target-26f57\" (UID: \"fb66d280-cf24-4549-8fdb-89f16b6080c1\") " pod="openshift-network-diagnostics/network-check-target-26f57" Oct 30 18:32:43.948275 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:43.948255 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 18:32:43.948340 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:43.948280 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 18:32:43.948340 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:43.948291 2575 projected.go:196] Error preparing data for projected volume kube-api-access-ztd9j for pod openshift-network-diagnostics/network-check-target-26f57: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 18:32:43.948419 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:43.948345 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fb66d280-cf24-4549-8fdb-89f16b6080c1-kube-api-access-ztd9j podName:fb66d280-cf24-4549-8fdb-89f16b6080c1 nodeName:}" failed. No retries permitted until 2026-10-30 18:32:44.948330232 +0000 UTC m=+3.167321634 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-ztd9j" (UniqueName: "kubernetes.io/projected/fb66d280-cf24-4549-8fdb-89f16b6080c1-kube-api-access-ztd9j") pod "network-check-target-26f57" (UID: "fb66d280-cf24-4549-8fdb-89f16b6080c1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 18:32:44.192622 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:44.192501 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 18:27:44 +0000 UTC" deadline="2028-06-30 05:45:10.97816377 +0000 UTC" Oct 30 18:32:44.192622 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:44.192573 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14603h12m26.785595535s" Oct 30 18:32:44.319606 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:44.318759 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-68r7k" event={"ID":"d2baf406-5501-4717-b5e1-d145edfe9ddc","Type":"ContainerStarted","Data":"f653dd2bb9642410489171060113e275c78337a3f25db513e81c1722e9966388"} Oct 30 18:32:44.322927 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:44.322886 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtz4g" event={"ID":"e91ad8e0-05ce-420f-9594-445f4d48739c","Type":"ContainerStarted","Data":"908390dea71dfceecda52d389bc886f06dadc8f7acb975b9402fe99fc931d605"} Oct 30 18:32:44.332937 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:44.332904 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qrdc6" event={"ID":"b0f929fc-a9ea-4935-9fe7-770c5e7d4de5","Type":"ContainerStarted","Data":"2df7bb0c79d3346dc1213c978fbd1bf6cf2feee082618ee2378c05df610217e3"} Oct 30 18:32:44.336057 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:44.336029 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bk5br" event={"ID":"05cad8b6-56f5-4bd1-b12d-b2ae456b7027","Type":"ContainerStarted","Data":"8ae7cf2fa0bd11e84801e93bc28861efbd30ad203a94848ef0f5199e314f4f4c"} Oct 30 18:32:44.338917 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:44.338886 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rpzvk" event={"ID":"eef45986-9475-4e3e-a3f9-fdb682474ff3","Type":"ContainerStarted","Data":"1cd2fe140d1ce2267185d56ec36bf3e19fb7cfc7194597c310311fe7067f8959"} Oct 30 18:32:44.341676 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:44.341648 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-163.us-east-2.compute.internal" event={"ID":"40b5c76fae82f6d7a38a1a745d5d3947","Type":"ContainerStarted","Data":"c9ae698e2ddbafabe23051c119ab3b29049da90efcd4e436acdec3095cd3fa73"} Oct 30 18:32:44.350430 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:44.349424 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4g8sn" event={"ID":"b4961ad6-2fc3-4c01-a565-6f66068d2958","Type":"ContainerStarted","Data":"56cf64efd808a743e34aa2da5f607cdce99f15da9da3be52befbd6612366c7d4"} Oct 30 18:32:44.353318 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:44.352625 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nl4m8" event={"ID":"5b826abe-3f79-43ab-8813-2406d5c4dae9","Type":"ContainerStarted","Data":"3fc6a6002c134d13eff65b44fca6ed477f75262fb065531496148ca5cc8559e7"} Oct 30 18:32:44.354844 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:44.354806 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-b876b" event={"ID":"ef86d9fa-a344-4886-b80c-89ce84f418ea","Type":"ContainerStarted","Data":"a2a9935a2a2d2b39234f40a39264586ed51525af2ed3b73ad1b2f5be53e581a7"} Oct 30 18:32:44.356575 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:44.356548 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-163.us-east-2.compute.internal" event={"ID":"3b404058373133d538844156a64ac398","Type":"ContainerStarted","Data":"172feea2873431f0fbd887a83aadcc45053c61f7631d004fbfbf5d3f0aa4dd77"} Oct 30 18:32:44.360315 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:44.360288 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" event={"ID":"81f52134-6a09-431f-9f80-412ae7bbead8","Type":"ContainerStarted","Data":"ebb7a6cb2f181cb9daf6993af1f5e8a539dc87cc81cd568fb2eb857afff8935c"} Oct 30 18:32:44.394498 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:44.394374 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"ovn-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:be9d8303ae4f79715ade971ddecdf6044ee6bb7b8d8b3a2f2f1e3bfe4e78c15a\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"ovn-acl-logging\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:be9d8303ae4f79715ade971ddecdf6044ee6bb7b8d8b3a2f2f1e3bfe4e78c15a\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"kube-rbac-proxy-node\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d017b7d199b1d54e889bb1eb6e572d8e6d94ad92610894009ee0562c2a5c0f51\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"kube-rbac-proxy-ovn-metrics\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d017b7d199b1d54e889bb1eb6e572d8e6d94ad92610894009ee0562c2a5c0f51\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"northd\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:be9d8303ae4f79715ade971ddecdf6044ee6bb7b8d8b3a2f2f1e3bfe4e78c15a\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"nbdb\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:be9d8303ae4f79715ade971ddecdf6044ee6bb7b8d8b3a2f2f1e3bfe4e78c15a\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"sbdb\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:be9d8303ae4f79715ade971ddecdf6044ee6bb7b8d8b3a2f2f1e3bfe4e78c15a\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"ovnkube-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:be9d8303ae4f79715ade971ddecdf6044ee6bb7b8d8b3a2f2f1e3bfe4e78c15a\\\": ErrImagePull: pull QPS exceeded\"]" pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" podUID="81f52134-6a09-431f-9f80-412ae7bbead8" Oct 30 18:32:44.642096 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:44.642013 2575 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 18:32:44.857075 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:44.856775 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e163025a-b19f-44e2-9058-b65f31bfbcc4-original-pull-secret\") pod \"global-pull-secret-syncer-zmdjg\" (UID: \"e163025a-b19f-44e2-9058-b65f31bfbcc4\") " pod="kube-system/global-pull-secret-syncer-zmdjg" Oct 30 18:32:44.857075 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:44.856856 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90488e60-cb70-4afa-a54b-6f333d345e84-metrics-certs\") pod \"network-metrics-daemon-dtppz\" (UID: \"90488e60-cb70-4afa-a54b-6f333d345e84\") " pod="openshift-multus/network-metrics-daemon-dtppz" Oct 30 18:32:44.857075 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:44.857027 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 18:32:44.857419 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:44.857099 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90488e60-cb70-4afa-a54b-6f333d345e84-metrics-certs podName:90488e60-cb70-4afa-a54b-6f333d345e84 nodeName:}" failed. No retries permitted until 2026-10-30 18:32:46.857078549 +0000 UTC m=+5.076069954 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/90488e60-cb70-4afa-a54b-6f333d345e84-metrics-certs") pod "network-metrics-daemon-dtppz" (UID: "90488e60-cb70-4afa-a54b-6f333d345e84") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 18:32:44.857419 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:44.857355 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 18:32:44.857530 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:44.857427 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e163025a-b19f-44e2-9058-b65f31bfbcc4-original-pull-secret podName:e163025a-b19f-44e2-9058-b65f31bfbcc4 nodeName:}" failed. No retries permitted until 2026-10-30 18:32:46.857407921 +0000 UTC m=+5.076399338 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e163025a-b19f-44e2-9058-b65f31bfbcc4-original-pull-secret") pod "global-pull-secret-syncer-zmdjg" (UID: "e163025a-b19f-44e2-9058-b65f31bfbcc4") : object "kube-system"/"original-pull-secret" not registered Oct 30 18:32:44.958672 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:44.958201 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztd9j\" (UniqueName: \"kubernetes.io/projected/fb66d280-cf24-4549-8fdb-89f16b6080c1-kube-api-access-ztd9j\") pod \"network-check-target-26f57\" (UID: \"fb66d280-cf24-4549-8fdb-89f16b6080c1\") " pod="openshift-network-diagnostics/network-check-target-26f57" Oct 30 18:32:44.958672 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:44.958439 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 18:32:44.958672 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:44.958461 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 18:32:44.958672 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:44.958476 2575 projected.go:196] Error preparing data for projected volume kube-api-access-ztd9j for pod openshift-network-diagnostics/network-check-target-26f57: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 18:32:44.958672 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:44.958539 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fb66d280-cf24-4549-8fdb-89f16b6080c1-kube-api-access-ztd9j podName:fb66d280-cf24-4549-8fdb-89f16b6080c1 nodeName:}" failed. No retries permitted until 2026-10-30 18:32:46.958520153 +0000 UTC m=+5.177511575 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-ztd9j" (UniqueName: "kubernetes.io/projected/fb66d280-cf24-4549-8fdb-89f16b6080c1-kube-api-access-ztd9j") pod "network-check-target-26f57" (UID: "fb66d280-cf24-4549-8fdb-89f16b6080c1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 18:32:45.193934 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:45.193745 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 18:27:44 +0000 UTC" deadline="2028-08-17 09:16:04.126186336 +0000 UTC" Oct 30 18:32:45.193934 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:45.193812 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15758h43m18.932378999s" Oct 30 18:32:45.303054 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:45.302960 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmdjg" Oct 30 18:32:45.303054 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:45.303055 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26f57" Oct 30 18:32:45.303332 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:45.303166 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26f57" podUID="fb66d280-cf24-4549-8fdb-89f16b6080c1" Oct 30 18:32:45.303672 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:45.303631 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dtppz" Oct 30 18:32:45.303819 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:45.303746 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dtppz" podUID="90488e60-cb70-4afa-a54b-6f333d345e84" Oct 30 18:32:45.303887 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:45.303851 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmdjg" podUID="e163025a-b19f-44e2-9058-b65f31bfbcc4" Oct 30 18:32:45.956732 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:45.956697 2575 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 18:32:46.874706 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:46.874648 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e163025a-b19f-44e2-9058-b65f31bfbcc4-original-pull-secret\") pod \"global-pull-secret-syncer-zmdjg\" (UID: \"e163025a-b19f-44e2-9058-b65f31bfbcc4\") " pod="kube-system/global-pull-secret-syncer-zmdjg" Oct 30 18:32:46.875124 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:46.874744 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90488e60-cb70-4afa-a54b-6f333d345e84-metrics-certs\") pod \"network-metrics-daemon-dtppz\" (UID: \"90488e60-cb70-4afa-a54b-6f333d345e84\") " pod="openshift-multus/network-metrics-daemon-dtppz" Oct 30 18:32:46.875124 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:46.874900 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 18:32:46.875124 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:46.874967 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90488e60-cb70-4afa-a54b-6f333d345e84-metrics-certs podName:90488e60-cb70-4afa-a54b-6f333d345e84 nodeName:}" failed. No retries permitted until 2026-10-30 18:32:50.874948674 +0000 UTC m=+9.093940079 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/90488e60-cb70-4afa-a54b-6f333d345e84-metrics-certs") pod "network-metrics-daemon-dtppz" (UID: "90488e60-cb70-4afa-a54b-6f333d345e84") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 18:32:46.875423 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:46.875401 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 18:32:46.875489 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:46.875464 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e163025a-b19f-44e2-9058-b65f31bfbcc4-original-pull-secret podName:e163025a-b19f-44e2-9058-b65f31bfbcc4 nodeName:}" failed. No retries permitted until 2026-10-30 18:32:50.875445812 +0000 UTC m=+9.094437230 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e163025a-b19f-44e2-9058-b65f31bfbcc4-original-pull-secret") pod "global-pull-secret-syncer-zmdjg" (UID: "e163025a-b19f-44e2-9058-b65f31bfbcc4") : object "kube-system"/"original-pull-secret" not registered Oct 30 18:32:46.975853 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:46.975467 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztd9j\" (UniqueName: \"kubernetes.io/projected/fb66d280-cf24-4549-8fdb-89f16b6080c1-kube-api-access-ztd9j\") pod \"network-check-target-26f57\" (UID: \"fb66d280-cf24-4549-8fdb-89f16b6080c1\") " pod="openshift-network-diagnostics/network-check-target-26f57" Oct 30 18:32:46.975853 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:46.975694 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 18:32:46.975853 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:46.975716 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 18:32:46.975853 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:46.975733 2575 projected.go:196] Error preparing data for projected volume kube-api-access-ztd9j for pod openshift-network-diagnostics/network-check-target-26f57: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 18:32:46.975853 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:46.975795 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fb66d280-cf24-4549-8fdb-89f16b6080c1-kube-api-access-ztd9j podName:fb66d280-cf24-4549-8fdb-89f16b6080c1 nodeName:}" failed. No retries permitted until 2026-10-30 18:32:50.9757734 +0000 UTC m=+9.194764829 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-ztd9j" (UniqueName: "kubernetes.io/projected/fb66d280-cf24-4549-8fdb-89f16b6080c1-kube-api-access-ztd9j") pod "network-check-target-26f57" (UID: "fb66d280-cf24-4549-8fdb-89f16b6080c1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 18:32:47.303377 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:47.303343 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dtppz" Oct 30 18:32:47.303573 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:47.303517 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dtppz" podUID="90488e60-cb70-4afa-a54b-6f333d345e84" Oct 30 18:32:47.303924 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:47.303903 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmdjg" Oct 30 18:32:47.304018 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:47.303992 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmdjg" podUID="e163025a-b19f-44e2-9058-b65f31bfbcc4" Oct 30 18:32:47.304069 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:47.304036 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26f57" Oct 30 18:32:47.304118 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:47.304081 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26f57" podUID="fb66d280-cf24-4549-8fdb-89f16b6080c1" Oct 30 18:32:47.391816 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:47.391771 2575 generic.go:356] "Generic (PLEG): container finished" podID="40b5c76fae82f6d7a38a1a745d5d3947" containerID="da595342be50d698960624c3f93468dbb0cf59f511a95cb11bdf08cff27fc501" exitCode=0 Oct 30 18:32:47.392001 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:47.391869 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-163.us-east-2.compute.internal" event={"ID":"40b5c76fae82f6d7a38a1a745d5d3947","Type":"ContainerDied","Data":"da595342be50d698960624c3f93468dbb0cf59f511a95cb11bdf08cff27fc501"} Oct 30 18:32:47.401482 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:47.401451 2575 generic.go:356] "Generic (PLEG): container finished" podID="5b826abe-3f79-43ab-8813-2406d5c4dae9" containerID="800031197ad01291a6da93b42297c55fa1d72fe479eb4dec9a83ab5f7622ce35" exitCode=0 Oct 30 18:32:47.401597 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:47.401510 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nl4m8" event={"ID":"5b826abe-3f79-43ab-8813-2406d5c4dae9","Type":"ContainerDied","Data":"800031197ad01291a6da93b42297c55fa1d72fe479eb4dec9a83ab5f7622ce35"} Oct 30 18:32:48.407999 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:48.407821 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-163.us-east-2.compute.internal" event={"ID":"40b5c76fae82f6d7a38a1a745d5d3947","Type":"ContainerStarted","Data":"603302b9d12c1d9fc525d9c3d19924af292b0e67809b0a7f2605a607abd8c559"} Oct 30 18:32:48.411066 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:48.411032 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4g8sn" event={"ID":"b4961ad6-2fc3-4c01-a565-6f66068d2958","Type":"ContainerStarted","Data":"e9b8c875b3cd6570dc0104286461d3825e365c93cf6e2b5fb614299ddfe4ecd0"} Oct 30 18:32:48.416989 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:48.416471 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-163.us-east-2.compute.internal" event={"ID":"3b404058373133d538844156a64ac398","Type":"ContainerStarted","Data":"3a368fef34a48d4c7a8d51a5a8cd6471b7d622da94accf45cfad71e78230ef1b"} Oct 30 18:32:48.466946 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:48.465725 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-163.us-east-2.compute.internal" podStartSLOduration=4.465705732 podStartE2EDuration="4.465705732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:32:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:32:48.43697384 +0000 UTC m=+6.655965267" watchObservedRunningTime="2026-10-30 18:32:48.465705732 +0000 UTC m=+6.684697175" Oct 30 18:32:48.466946 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:48.466461 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-163.us-east-2.compute.internal" podStartSLOduration=4.466447221 podStartE2EDuration="4.466447221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:32:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:32:48.466381787 +0000 UTC m=+6.685373214" watchObservedRunningTime="2026-10-30 18:32:48.466447221 +0000 UTC m=+6.685438647" Oct 30 18:32:48.497199 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:48.496377 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-4g8sn" podStartSLOduration=1.4753864110000001 podStartE2EDuration="5.496358465s" totalImagesPullingTime="4.020972054s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:32:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:32:48.496039131 +0000 UTC m=+6.715030558" watchObservedRunningTime="2026-10-30 18:32:48.496358465 +0000 UTC m=+6.715349893" Oct 30 18:32:49.303256 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:49.303227 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dtppz" Oct 30 18:32:49.303413 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:49.303374 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dtppz" podUID="90488e60-cb70-4afa-a54b-6f333d345e84" Oct 30 18:32:49.303488 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:49.303457 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmdjg" Oct 30 18:32:49.303557 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:49.303536 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmdjg" podUID="e163025a-b19f-44e2-9058-b65f31bfbcc4" Oct 30 18:32:49.303606 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:49.303599 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26f57" Oct 30 18:32:49.303689 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:49.303669 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26f57" podUID="fb66d280-cf24-4549-8fdb-89f16b6080c1" Oct 30 18:32:49.421632 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:49.421531 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtz4g" event={"ID":"e91ad8e0-05ce-420f-9594-445f4d48739c","Type":"ContainerStarted","Data":"a23a2eb65b01ac390c4f9ac10a6abebcccae8fd32e514712b8aa868787897735"} Oct 30 18:32:49.424313 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:49.424273 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qrdc6" event={"ID":"b0f929fc-a9ea-4935-9fe7-770c5e7d4de5","Type":"ContainerStarted","Data":"a4db0b09217919641632700cfffb7aa2887d66e01ec4ae30770e1e0c5a57c849"} Oct 30 18:32:49.443974 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:49.443912 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-qrdc6" podStartSLOduration=1.404398835 podStartE2EDuration="6.443893246s" totalImagesPullingTime="5.039494411s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:32:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:32:49.443009381 +0000 UTC m=+7.662000801" watchObservedRunningTime="2026-10-30 18:32:49.443893246 +0000 UTC m=+7.662884672" Oct 30 18:32:50.915686 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:50.915626 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90488e60-cb70-4afa-a54b-6f333d345e84-metrics-certs\") pod \"network-metrics-daemon-dtppz\" (UID: \"90488e60-cb70-4afa-a54b-6f333d345e84\") " pod="openshift-multus/network-metrics-daemon-dtppz" Oct 30 18:32:50.916117 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:50.915714 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e163025a-b19f-44e2-9058-b65f31bfbcc4-original-pull-secret\") pod \"global-pull-secret-syncer-zmdjg\" (UID: \"e163025a-b19f-44e2-9058-b65f31bfbcc4\") " pod="kube-system/global-pull-secret-syncer-zmdjg" Oct 30 18:32:50.916117 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:50.915851 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 18:32:50.916117 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:50.915909 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e163025a-b19f-44e2-9058-b65f31bfbcc4-original-pull-secret podName:e163025a-b19f-44e2-9058-b65f31bfbcc4 nodeName:}" failed. No retries permitted until 2026-10-30 18:32:58.915889137 +0000 UTC m=+17.134880554 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e163025a-b19f-44e2-9058-b65f31bfbcc4-original-pull-secret") pod "global-pull-secret-syncer-zmdjg" (UID: "e163025a-b19f-44e2-9058-b65f31bfbcc4") : object "kube-system"/"original-pull-secret" not registered Oct 30 18:32:50.916334 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:50.916313 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 18:32:50.916399 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:50.916359 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90488e60-cb70-4afa-a54b-6f333d345e84-metrics-certs podName:90488e60-cb70-4afa-a54b-6f333d345e84 nodeName:}" failed. No retries permitted until 2026-10-30 18:32:58.916346964 +0000 UTC m=+17.135338380 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/90488e60-cb70-4afa-a54b-6f333d345e84-metrics-certs") pod "network-metrics-daemon-dtppz" (UID: "90488e60-cb70-4afa-a54b-6f333d345e84") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 18:32:50.956980 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:50.956926 2575 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 18:32:51.016262 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:51.016132 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztd9j\" (UniqueName: \"kubernetes.io/projected/fb66d280-cf24-4549-8fdb-89f16b6080c1-kube-api-access-ztd9j\") pod \"network-check-target-26f57\" (UID: \"fb66d280-cf24-4549-8fdb-89f16b6080c1\") " pod="openshift-network-diagnostics/network-check-target-26f57" Oct 30 18:32:51.016457 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:51.016343 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 18:32:51.016457 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:51.016372 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 18:32:51.016457 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:51.016387 2575 projected.go:196] Error preparing data for projected volume kube-api-access-ztd9j for pod openshift-network-diagnostics/network-check-target-26f57: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 18:32:51.016457 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:51.016448 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fb66d280-cf24-4549-8fdb-89f16b6080c1-kube-api-access-ztd9j podName:fb66d280-cf24-4549-8fdb-89f16b6080c1 nodeName:}" failed. No retries permitted until 2026-10-30 18:32:59.016431163 +0000 UTC m=+17.235422579 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-ztd9j" (UniqueName: "kubernetes.io/projected/fb66d280-cf24-4549-8fdb-89f16b6080c1-kube-api-access-ztd9j") pod "network-check-target-26f57" (UID: "fb66d280-cf24-4549-8fdb-89f16b6080c1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 18:32:51.203356 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:51.203213 2575 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T18:32:50.956948424Z","UUID":"5c432555-4c75-4104-a13c-cc6b0f0fe468","Handler":null,"Name":"","Endpoint":""} Oct 30 18:32:51.205227 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:51.205200 2575 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 18:32:51.205363 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:51.205253 2575 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 18:32:51.302795 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:51.302763 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dtppz" Oct 30 18:32:51.302996 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:51.302820 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26f57" Oct 30 18:32:51.302996 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:51.302770 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmdjg" Oct 30 18:32:51.302996 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:51.302915 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dtppz" podUID="90488e60-cb70-4afa-a54b-6f333d345e84" Oct 30 18:32:51.303149 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:51.303031 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmdjg" podUID="e163025a-b19f-44e2-9058-b65f31bfbcc4" Oct 30 18:32:51.303263 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:51.303230 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26f57" podUID="fb66d280-cf24-4549-8fdb-89f16b6080c1" Oct 30 18:32:51.430820 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:51.430789 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtz4g" event={"ID":"e91ad8e0-05ce-420f-9594-445f4d48739c","Type":"ContainerStarted","Data":"7e59d369a8d6a5794be60d1b04be33f604a1442f483f2de81e6254bac32a6b17"} Oct 30 18:32:51.435647 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:51.435619 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-68r7k" event={"ID":"d2baf406-5501-4717-b5e1-d145edfe9ddc","Type":"ContainerStarted","Data":"92c602b2ace40209e3793984418222934b6f70c9f76f9ca5c0f223cf882802b6"} Oct 30 18:32:51.472261 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:51.469413 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-68r7k" podStartSLOduration=1.234980114 podStartE2EDuration="8.469384321s" totalImagesPullingTime="7.234404207s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:32:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:32:51.468335975 +0000 UTC m=+9.687327442" watchObservedRunningTime="2026-10-30 18:32:51.469384321 +0000 UTC m=+9.688375748" Oct 30 18:32:51.663554 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:51.663510 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:51.784460 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:51.784412 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:52.097171 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:52.097072 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:52.214732 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:52.214689 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:52.440591 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:52.440493 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtz4g" event={"ID":"e91ad8e0-05ce-420f-9594-445f4d48739c","Type":"ContainerStarted","Data":"d37eea440c72121e53cb9e4ade9f4132c5cc0187949933c31ea0342810cd190d"} Oct 30 18:32:52.442119 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:52.442085 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bk5br" event={"ID":"05cad8b6-56f5-4bd1-b12d-b2ae456b7027","Type":"ContainerStarted","Data":"c5a9f1d46e862e66cd6f18516f3edc7581e40516d9f02a726fdfbc742de9508a"} Oct 30 18:32:52.444156 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:52.444117 2575 generic.go:356] "Generic (PLEG): container finished" podID="5b826abe-3f79-43ab-8813-2406d5c4dae9" containerID="7844f15920df2b73e16553ce0bae91c18981aed2246e8859c491973928bc3dc0" exitCode=0 Oct 30 18:32:52.444310 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:52.444168 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nl4m8" event={"ID":"5b826abe-3f79-43ab-8813-2406d5c4dae9","Type":"ContainerDied","Data":"7844f15920df2b73e16553ce0bae91c18981aed2246e8859c491973928bc3dc0"} Oct 30 18:32:52.465617 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:52.465573 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtz4g" podStartSLOduration=1.428388142 podStartE2EDuration="9.4655596s" totalImagesPullingTime="8.037171458s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:32:43 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:32:52.465016401 +0000 UTC m=+10.684007827" watchObservedRunningTime="2026-10-30 18:32:52.4655596 +0000 UTC m=+10.684551025" Oct 30 18:32:52.511765 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:52.511720 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-bk5br" podStartSLOduration=1.692545279 podStartE2EDuration="9.511704009s" totalImagesPullingTime="7.81915873s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:32:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:32:52.511375162 +0000 UTC m=+10.730366587" watchObservedRunningTime="2026-10-30 18:32:52.511704009 +0000 UTC m=+10.730695435" Oct 30 18:32:53.143777 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:53.143543 2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-4g8sn" Oct 30 18:32:53.144269 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:53.144248 2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-4g8sn" Oct 30 18:32:53.302748 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:53.302725 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dtppz" Oct 30 18:32:53.302844 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:53.302828 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dtppz" podUID="90488e60-cb70-4afa-a54b-6f333d345e84" Oct 30 18:32:53.302895 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:53.302884 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmdjg" Oct 30 18:32:53.302953 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:53.302931 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmdjg" podUID="e163025a-b19f-44e2-9058-b65f31bfbcc4" Oct 30 18:32:53.302987 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:53.302980 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26f57" Oct 30 18:32:53.303037 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:53.303024 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26f57" podUID="fb66d280-cf24-4549-8fdb-89f16b6080c1" Oct 30 18:32:53.447910 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:53.447847 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rpzvk" event={"ID":"eef45986-9475-4e3e-a3f9-fdb682474ff3","Type":"ContainerStarted","Data":"f25ec2598738dd3b7dbe37897fa9862cb2cdaf8241258a03980e0ede6d49f90c"} Oct 30 18:32:53.450197 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:53.450168 2575 generic.go:356] "Generic (PLEG): container finished" podID="5b826abe-3f79-43ab-8813-2406d5c4dae9" containerID="bf11d46cd8e7eaa012e2192c2590c0a17692a1a734704c7c1ba63b5fe9740e23" exitCode=0 Oct 30 18:32:53.450327 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:53.450245 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nl4m8" event={"ID":"5b826abe-3f79-43ab-8813-2406d5c4dae9","Type":"ContainerDied","Data":"bf11d46cd8e7eaa012e2192c2590c0a17692a1a734704c7c1ba63b5fe9740e23"} Oct 30 18:32:53.451818 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:53.451756 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-b876b" event={"ID":"ef86d9fa-a344-4886-b80c-89ce84f418ea","Type":"ContainerStarted","Data":"2e9def72da94792176bf02e3687e1e2caf679a8bd8eaad69c0a687f070778451"} Oct 30 18:32:53.452191 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:53.452137 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-4g8sn" Oct 30 18:32:53.452738 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:53.452724 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-4g8sn" Oct 30 18:32:53.490636 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:53.490589 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-rpzvk" podStartSLOduration=1.214998505 podStartE2EDuration="10.490577079s" totalImagesPullingTime="9.275578574s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:32:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:32:53.471755932 +0000 UTC m=+11.690747357" watchObservedRunningTime="2026-10-30 18:32:53.490577079 +0000 UTC m=+11.709568504" Oct 30 18:32:53.490928 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:53.490899 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-b876b" podStartSLOduration=2.990721778 podStartE2EDuration="10.490892771s" totalImagesPullingTime="7.500170993s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:32:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:32:53.49041159 +0000 UTC m=+11.709403015" watchObservedRunningTime="2026-10-30 18:32:53.490892771 +0000 UTC m=+11.709884197" Oct 30 18:32:53.639936 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:53.639903 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:53.768081 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:53.768010 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:53.889358 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:53.889320 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:54.456126 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:54.456095 2575 generic.go:356] "Generic (PLEG): container finished" podID="5b826abe-3f79-43ab-8813-2406d5c4dae9" containerID="173a62de47c579d5b1ddd71171e22fb3ca597f987d7a14ecd9805e1200e2a7ed" exitCode=0 Oct 30 18:32:54.456698 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:54.456202 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nl4m8" event={"ID":"5b826abe-3f79-43ab-8813-2406d5c4dae9","Type":"ContainerDied","Data":"173a62de47c579d5b1ddd71171e22fb3ca597f987d7a14ecd9805e1200e2a7ed"} Oct 30 18:32:54.898282 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:54.898245 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:55.032443 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:55.032410 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:55.302446 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:55.302364 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dtppz" Oct 30 18:32:55.302600 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:55.302381 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmdjg" Oct 30 18:32:55.302600 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:55.302505 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dtppz" podUID="90488e60-cb70-4afa-a54b-6f333d345e84" Oct 30 18:32:55.302600 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:55.302534 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26f57" Oct 30 18:32:55.302754 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:55.302616 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmdjg" podUID="e163025a-b19f-44e2-9058-b65f31bfbcc4" Oct 30 18:32:55.302754 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:55.302713 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26f57" podUID="fb66d280-cf24-4549-8fdb-89f16b6080c1" Oct 30 18:32:55.449731 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:55.449699 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:55.591586 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:55.591507 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:55.737790 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:55.737752 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:56.462725 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:56.462474 2575 generic.go:356] "Generic (PLEG): container finished" podID="5b826abe-3f79-43ab-8813-2406d5c4dae9" containerID="e3060fff28d6443c7e4648c778be4126c6732ff40d8143db05f6a0db6b4ef4c3" exitCode=0 Oct 30 18:32:56.462725 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:56.462544 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nl4m8" event={"ID":"5b826abe-3f79-43ab-8813-2406d5c4dae9","Type":"ContainerDied","Data":"e3060fff28d6443c7e4648c778be4126c6732ff40d8143db05f6a0db6b4ef4c3"} Oct 30 18:32:56.955678 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:56.955597 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:57.253645 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:57.253574 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:57.303082 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:57.303052 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26f57" Oct 30 18:32:57.303337 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:57.303305 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26f57" podUID="fb66d280-cf24-4549-8fdb-89f16b6080c1" Oct 30 18:32:57.303389 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:57.303336 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmdjg" Oct 30 18:32:57.303434 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:57.303414 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmdjg" podUID="e163025a-b19f-44e2-9058-b65f31bfbcc4" Oct 30 18:32:57.303493 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:57.303478 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dtppz" Oct 30 18:32:57.303591 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:57.303565 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dtppz" podUID="90488e60-cb70-4afa-a54b-6f333d345e84" Oct 30 18:32:57.431235 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:57.431191 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:57.467424 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:57.467397 2575 generic.go:356] "Generic (PLEG): container finished" podID="5b826abe-3f79-43ab-8813-2406d5c4dae9" containerID="c246758d782ae07e82f94a156c6007334a401ef34106ea12fe38278878b79901" exitCode=0 Oct 30 18:32:57.467518 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:57.467448 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nl4m8" event={"ID":"5b826abe-3f79-43ab-8813-2406d5c4dae9","Type":"ContainerDied","Data":"c246758d782ae07e82f94a156c6007334a401ef34106ea12fe38278878b79901"} Oct 30 18:32:57.559488 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:57.559458 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:57.769268 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:57.769197 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:58.138649 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:58.138524 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:58.216581 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:58.216543 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bk5br_05cad8b6-56f5-4bd1-b12d-b2ae456b7027/dns-node-resolver/0.log" Oct 30 18:32:58.476084 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:58.475997 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nl4m8" event={"ID":"5b826abe-3f79-43ab-8813-2406d5c4dae9","Type":"ContainerStarted","Data":"cfae00cd67a7d6b788b52f274d4e2a70bfec891a0e70171a45d1f78ff9930855"} Oct 30 18:32:58.505966 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:58.505908 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-nl4m8" podStartSLOduration=5.525385656 podStartE2EDuration="15.505891229s" totalImagesPullingTime="9.980505573s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:32:43 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:32:58.505738685 +0000 UTC m=+16.724730108" watchObservedRunningTime="2026-10-30 18:32:58.505891229 +0000 UTC m=+16.724882678" Oct 30 18:32:58.542069 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:58.542037 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:58.749698 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:58.749563 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:58.984563 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:58.984529 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90488e60-cb70-4afa-a54b-6f333d345e84-metrics-certs\") pod \"network-metrics-daemon-dtppz\" (UID: \"90488e60-cb70-4afa-a54b-6f333d345e84\") " pod="openshift-multus/network-metrics-daemon-dtppz" Oct 30 18:32:58.984758 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:58.984600 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e163025a-b19f-44e2-9058-b65f31bfbcc4-original-pull-secret\") pod \"global-pull-secret-syncer-zmdjg\" (UID: \"e163025a-b19f-44e2-9058-b65f31bfbcc4\") " pod="kube-system/global-pull-secret-syncer-zmdjg" Oct 30 18:32:58.984758 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:58.984725 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 18:32:58.984837 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:58.984772 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e163025a-b19f-44e2-9058-b65f31bfbcc4-original-pull-secret podName:e163025a-b19f-44e2-9058-b65f31bfbcc4 nodeName:}" failed. No retries permitted until 2026-10-30 18:33:14.984760552 +0000 UTC m=+33.203751966 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e163025a-b19f-44e2-9058-b65f31bfbcc4-original-pull-secret") pod "global-pull-secret-syncer-zmdjg" (UID: "e163025a-b19f-44e2-9058-b65f31bfbcc4") : object "kube-system"/"original-pull-secret" not registered Oct 30 18:32:58.985212 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:58.985193 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 18:32:58.985292 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:58.985252 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90488e60-cb70-4afa-a54b-6f333d345e84-metrics-certs podName:90488e60-cb70-4afa-a54b-6f333d345e84 nodeName:}" failed. No retries permitted until 2026-10-30 18:33:14.985236738 +0000 UTC m=+33.204228156 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/90488e60-cb70-4afa-a54b-6f333d345e84-metrics-certs") pod "network-metrics-daemon-dtppz" (UID: "90488e60-cb70-4afa-a54b-6f333d345e84") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 18:32:58.989320 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:58.989295 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:59.085652 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:59.085581 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztd9j\" (UniqueName: \"kubernetes.io/projected/fb66d280-cf24-4549-8fdb-89f16b6080c1-kube-api-access-ztd9j\") pod \"network-check-target-26f57\" (UID: \"fb66d280-cf24-4549-8fdb-89f16b6080c1\") " pod="openshift-network-diagnostics/network-check-target-26f57" Oct 30 18:32:59.085805 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:59.085776 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 18:32:59.085805 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:59.085796 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 18:32:59.085805 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:59.085807 2575 projected.go:196] Error preparing data for projected volume kube-api-access-ztd9j for pod openshift-network-diagnostics/network-check-target-26f57: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 18:32:59.085907 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:59.085865 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fb66d280-cf24-4549-8fdb-89f16b6080c1-kube-api-access-ztd9j podName:fb66d280-cf24-4549-8fdb-89f16b6080c1 nodeName:}" failed. No retries permitted until 2026-10-30 18:33:15.085847082 +0000 UTC m=+33.304838506 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-ztd9j" (UniqueName: "kubernetes.io/projected/fb66d280-cf24-4549-8fdb-89f16b6080c1-kube-api-access-ztd9j") pod "network-check-target-26f57" (UID: "fb66d280-cf24-4549-8fdb-89f16b6080c1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 18:32:59.167230 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:59.167192 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:59.302936 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:59.302898 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26f57" Oct 30 18:32:59.302936 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:59.302925 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dtppz" Oct 30 18:32:59.303173 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:59.303022 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmdjg" Oct 30 18:32:59.303173 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:59.303044 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26f57" podUID="fb66d280-cf24-4549-8fdb-89f16b6080c1" Oct 30 18:32:59.303173 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:59.303100 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmdjg" podUID="e163025a-b19f-44e2-9058-b65f31bfbcc4" Oct 30 18:32:59.303332 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:32:59.303183 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dtppz" podUID="90488e60-cb70-4afa-a54b-6f333d345e84" Oct 30 18:32:59.555087 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:59.555015 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:59.621213 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:32:59.621188 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qrdc6_b0f929fc-a9ea-4935-9fe7-770c5e7d4de5/node-ca/0.log" Oct 30 18:33:00.336425 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:00.336387 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:00.477230 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:00.477199 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:00.590073 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:00.589992 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:00.699101 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:00.699065 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:00.829349 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:00.829314 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:00.961512 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:00.961443 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:01.137533 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:01.137475 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:01.303135 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:01.303057 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmdjg" Oct 30 18:33:01.303303 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:01.303057 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dtppz" Oct 30 18:33:01.303303 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:33:01.303184 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmdjg" podUID="e163025a-b19f-44e2-9058-b65f31bfbcc4" Oct 30 18:33:01.303303 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:01.303056 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26f57" Oct 30 18:33:01.303454 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:33:01.303314 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26f57" podUID="fb66d280-cf24-4549-8fdb-89f16b6080c1" Oct 30 18:33:01.303454 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:33:01.303252 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dtppz" podUID="90488e60-cb70-4afa-a54b-6f333d345e84" Oct 30 18:33:01.308934 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:01.308897 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:01.432085 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:01.432053 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:01.547859 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:01.547812 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:01.704201 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:01.704105 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:01.821429 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:01.821403 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:01.922363 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:01.922319 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:02.145454 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:02.145378 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:02.265410 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:02.265376 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:02.410986 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:02.410912 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:02.726162 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:02.726069 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:02.937343 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:02.937314 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:03.047979 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:03.047899 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:03.232545 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:03.232510 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:03.303247 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:03.303165 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dtppz" Oct 30 18:33:03.303387 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:03.303165 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmdjg" Oct 30 18:33:03.303387 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:33:03.303290 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dtppz" podUID="90488e60-cb70-4afa-a54b-6f333d345e84" Oct 30 18:33:03.303387 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:33:03.303347 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmdjg" podUID="e163025a-b19f-44e2-9058-b65f31bfbcc4" Oct 30 18:33:03.303387 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:03.303173 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26f57" Oct 30 18:33:03.303566 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:33:03.303423 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26f57" podUID="fb66d280-cf24-4549-8fdb-89f16b6080c1" Oct 30 18:33:03.394037 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:03.394001 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:03.541424 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:03.541393 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:04.101573 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:04.101537 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:04.224661 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:04.224618 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:04.341924 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:04.341890 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:04.734603 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:04.734568 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:04.966821 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:04.966786 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:05.303041 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:05.302535 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dtppz" Oct 30 18:33:05.303041 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:05.302582 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26f57" Oct 30 18:33:05.303041 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:05.302665 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmdjg" Oct 30 18:33:05.303041 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:33:05.302685 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dtppz" podUID="90488e60-cb70-4afa-a54b-6f333d345e84" Oct 30 18:33:05.303041 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:33:05.302764 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmdjg" podUID="e163025a-b19f-44e2-9058-b65f31bfbcc4" Oct 30 18:33:05.303041 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:33:05.302825 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26f57" podUID="fb66d280-cf24-4549-8fdb-89f16b6080c1" Oct 30 18:33:05.345496 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:05.345459 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:05.538436 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:05.538391 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:05.828798 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:05.828774 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:05.937810 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:05.937661 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:06.142196 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:06.142163 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:06.495911 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:06.495836 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" event={"ID":"81f52134-6a09-431f-9f80-412ae7bbead8","Type":"ContainerStarted","Data":"ffd36bd0ffc90de7895ee94f4bd263794936252e7b1ed6c21cb98f1fb22a06a7"} Oct 30 18:33:06.495911 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:06.495896 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" event={"ID":"81f52134-6a09-431f-9f80-412ae7bbead8","Type":"ContainerStarted","Data":"fed3a9c5dc42f4238a5654ee6baa88d99a4e5875cf08e5a6a175228d108a37b7"} Oct 30 18:33:06.496377 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:06.495925 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" event={"ID":"81f52134-6a09-431f-9f80-412ae7bbead8","Type":"ContainerStarted","Data":"7f14f9b1bb2dfa24dd51b9cf954ebdae59ca03c72c395d0e034a027ff57d9560"} Oct 30 18:33:06.496377 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:06.495955 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" event={"ID":"81f52134-6a09-431f-9f80-412ae7bbead8","Type":"ContainerStarted","Data":"f9e6403b632d49b72055638cc0737166f5c3631ed2b3e00216a0a11f104364f5"} Oct 30 18:33:06.496377 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:06.495983 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" event={"ID":"81f52134-6a09-431f-9f80-412ae7bbead8","Type":"ContainerStarted","Data":"03b499f8f1b652c2d8e9aad7302ed0a0e6db0973ec654c84cb15cced42fda54e"} Oct 30 18:33:06.496377 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:06.496009 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" event={"ID":"81f52134-6a09-431f-9f80-412ae7bbead8","Type":"ContainerStarted","Data":"7dd16d5dc79db02d23b1c85463f8cd8dcccc1da1cb76ee22ae90c621417f1ab8"} Oct 30 18:33:06.747658 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:06.747582 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:06.929671 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:06.929643 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:07.302517 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:07.302495 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmdjg" Oct 30 18:33:07.302668 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:07.302613 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26f57" Oct 30 18:33:07.302724 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:33:07.302608 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmdjg" podUID="e163025a-b19f-44e2-9058-b65f31bfbcc4" Oct 30 18:33:07.302724 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:33:07.302692 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26f57" podUID="fb66d280-cf24-4549-8fdb-89f16b6080c1" Oct 30 18:33:07.302724 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:07.302704 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dtppz" Oct 30 18:33:07.302836 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:33:07.302766 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dtppz" podUID="90488e60-cb70-4afa-a54b-6f333d345e84" Oct 30 18:33:07.322762 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:07.322716 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:07.527726 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:07.527700 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:08.129993 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:08.129970 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:08.328246 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:08.328219 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:08.502114 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:08.502036 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" event={"ID":"81f52134-6a09-431f-9f80-412ae7bbead8","Type":"ContainerStarted","Data":"93f7b46a8e8183ffeac639d84aa3e60c60522a0cee8b7f598b623787a8b64e09"} Oct 30 18:33:08.526850 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:08.526826 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:09.302705 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:09.302676 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dtppz" Oct 30 18:33:09.303208 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:09.302740 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26f57" Oct 30 18:33:09.303208 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:09.302766 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmdjg" Oct 30 18:33:09.303208 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:33:09.302843 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26f57" podUID="fb66d280-cf24-4549-8fdb-89f16b6080c1" Oct 30 18:33:09.303208 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:33:09.302908 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmdjg" podUID="e163025a-b19f-44e2-9058-b65f31bfbcc4" Oct 30 18:33:09.303208 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:33:09.302918 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dtppz" podUID="90488e60-cb70-4afa-a54b-6f333d345e84" Oct 30 18:33:10.509518 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:10.509439 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" event={"ID":"81f52134-6a09-431f-9f80-412ae7bbead8","Type":"ContainerStarted","Data":"953f6436a065d295bfd346735fe4abb8eb2b02d381f4d2e7b5336781a5ea926c"} Oct 30 18:33:10.510275 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:10.509700 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:33:10.510275 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:10.509777 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:33:10.510275 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:10.509828 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:33:10.526002 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:10.525978 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:33:10.526122 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:10.526069 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:33:10.547834 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:10.547789 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" podStartSLOduration=5.286512257 podStartE2EDuration="27.547776532s" totalImagesPullingTime="22.261264275s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:32:43 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:33:10.547613056 +0000 UTC m=+28.766604478" watchObservedRunningTime="2026-10-30 18:33:10.547776532 +0000 UTC m=+28.766767953" Oct 30 18:33:11.024866 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:11.024824 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:11.129871 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:11.129794 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:11.302778 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:11.302739 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmdjg" Oct 30 18:33:11.302778 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:11.302780 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26f57" Oct 30 18:33:11.302988 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:11.302865 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dtppz" Oct 30 18:33:11.302988 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:33:11.302873 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmdjg" podUID="e163025a-b19f-44e2-9058-b65f31bfbcc4" Oct 30 18:33:11.302988 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:33:11.302929 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26f57" podUID="fb66d280-cf24-4549-8fdb-89f16b6080c1" Oct 30 18:33:11.303210 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:33:11.303037 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dtppz" podUID="90488e60-cb70-4afa-a54b-6f333d345e84" Oct 30 18:33:13.302670 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:13.302481 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dtppz" Oct 30 18:33:13.303131 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:13.302496 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmdjg" Oct 30 18:33:13.303131 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:33:13.302753 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dtppz" podUID="90488e60-cb70-4afa-a54b-6f333d345e84" Oct 30 18:33:13.303131 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:13.302525 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26f57" Oct 30 18:33:13.303131 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:33:13.302835 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmdjg" podUID="e163025a-b19f-44e2-9058-b65f31bfbcc4" Oct 30 18:33:13.303131 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:33:13.302902 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26f57" podUID="fb66d280-cf24-4549-8fdb-89f16b6080c1" Oct 30 18:33:15.002647 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:15.002617 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90488e60-cb70-4afa-a54b-6f333d345e84-metrics-certs\") pod \"network-metrics-daemon-dtppz\" (UID: \"90488e60-cb70-4afa-a54b-6f333d345e84\") " pod="openshift-multus/network-metrics-daemon-dtppz" Oct 30 18:33:15.003101 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:15.002681 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e163025a-b19f-44e2-9058-b65f31bfbcc4-original-pull-secret\") pod \"global-pull-secret-syncer-zmdjg\" (UID: \"e163025a-b19f-44e2-9058-b65f31bfbcc4\") " pod="kube-system/global-pull-secret-syncer-zmdjg" Oct 30 18:33:15.003101 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:33:15.002755 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 18:33:15.003101 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:33:15.002784 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 18:33:15.003101 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:33:15.002821 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90488e60-cb70-4afa-a54b-6f333d345e84-metrics-certs podName:90488e60-cb70-4afa-a54b-6f333d345e84 nodeName:}" failed. No retries permitted until 2026-10-30 18:33:47.002806474 +0000 UTC m=+65.221797893 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/90488e60-cb70-4afa-a54b-6f333d345e84-metrics-certs") pod "network-metrics-daemon-dtppz" (UID: "90488e60-cb70-4afa-a54b-6f333d345e84") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 18:33:15.003101 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:33:15.002843 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e163025a-b19f-44e2-9058-b65f31bfbcc4-original-pull-secret podName:e163025a-b19f-44e2-9058-b65f31bfbcc4 nodeName:}" failed. No retries permitted until 2026-10-30 18:33:47.002826687 +0000 UTC m=+65.221818090 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e163025a-b19f-44e2-9058-b65f31bfbcc4-original-pull-secret") pod "global-pull-secret-syncer-zmdjg" (UID: "e163025a-b19f-44e2-9058-b65f31bfbcc4") : object "kube-system"/"original-pull-secret" not registered Oct 30 18:33:15.103601 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:15.103575 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztd9j\" (UniqueName: \"kubernetes.io/projected/fb66d280-cf24-4549-8fdb-89f16b6080c1-kube-api-access-ztd9j\") pod \"network-check-target-26f57\" (UID: \"fb66d280-cf24-4549-8fdb-89f16b6080c1\") " pod="openshift-network-diagnostics/network-check-target-26f57" Oct 30 18:33:15.103765 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:33:15.103734 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 18:33:15.103765 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:33:15.103751 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 18:33:15.103765 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:33:15.103760 2575 projected.go:196] Error preparing data for projected volume kube-api-access-ztd9j for pod openshift-network-diagnostics/network-check-target-26f57: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 18:33:15.103854 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:33:15.103807 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fb66d280-cf24-4549-8fdb-89f16b6080c1-kube-api-access-ztd9j podName:fb66d280-cf24-4549-8fdb-89f16b6080c1 nodeName:}" failed. No retries permitted until 2026-10-30 18:33:47.103792172 +0000 UTC m=+65.322783589 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-ztd9j" (UniqueName: "kubernetes.io/projected/fb66d280-cf24-4549-8fdb-89f16b6080c1-kube-api-access-ztd9j") pod "network-check-target-26f57" (UID: "fb66d280-cf24-4549-8fdb-89f16b6080c1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 18:33:15.302813 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:15.302733 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmdjg" Oct 30 18:33:15.302971 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:15.302854 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dtppz" Oct 30 18:33:15.302971 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:33:15.302855 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmdjg" podUID="e163025a-b19f-44e2-9058-b65f31bfbcc4" Oct 30 18:33:15.302971 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:15.302878 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26f57" Oct 30 18:33:15.302971 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:33:15.302946 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dtppz" podUID="90488e60-cb70-4afa-a54b-6f333d345e84" Oct 30 18:33:15.303191 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:33:15.303001 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26f57" podUID="fb66d280-cf24-4549-8fdb-89f16b6080c1" Oct 30 18:33:16.037969 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:16.037932 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:16.143780 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:16.143751 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:16.318658 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:16.318623 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:16.423674 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:16.423640 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:17.302486 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:17.302447 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dtppz" Oct 30 18:33:17.302982 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:17.302527 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26f57" Oct 30 18:33:17.302982 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:17.302550 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmdjg" Oct 30 18:33:17.302982 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:33:17.302634 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dtppz" podUID="90488e60-cb70-4afa-a54b-6f333d345e84" Oct 30 18:33:17.302982 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:33:17.302718 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmdjg" podUID="e163025a-b19f-44e2-9058-b65f31bfbcc4" Oct 30 18:33:17.302982 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:33:17.302816 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26f57" podUID="fb66d280-cf24-4549-8fdb-89f16b6080c1" Oct 30 18:33:17.511992 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:17.511960 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:17.643116 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:17.643083 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:17.835966 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:17.835938 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:17.960002 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:17.959943 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:18.096469 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:18.096442 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:19.102984 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:19.102902 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zmdjg"] Oct 30 18:33:19.103484 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:19.103033 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmdjg" Oct 30 18:33:19.103484 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:33:19.103161 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmdjg" podUID="e163025a-b19f-44e2-9058-b65f31bfbcc4" Oct 30 18:33:19.105952 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:19.105586 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-26f57"] Oct 30 18:33:19.105952 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:19.105692 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26f57" Oct 30 18:33:19.105952 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:33:19.105789 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26f57" podUID="fb66d280-cf24-4549-8fdb-89f16b6080c1" Oct 30 18:33:19.106378 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:19.106354 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dtppz"] Oct 30 18:33:19.106500 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:19.106449 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dtppz" Oct 30 18:33:19.106571 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:33:19.106551 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dtppz" podUID="90488e60-cb70-4afa-a54b-6f333d345e84" Oct 30 18:33:20.303178 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:20.303078 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dtppz" Oct 30 18:33:20.303664 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:33:20.303214 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dtppz" podUID="90488e60-cb70-4afa-a54b-6f333d345e84" Oct 30 18:33:21.302539 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:21.302505 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmdjg" Oct 30 18:33:21.302539 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:21.302538 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26f57" Oct 30 18:33:21.302790 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:33:21.302608 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmdjg" podUID="e163025a-b19f-44e2-9058-b65f31bfbcc4" Oct 30 18:33:21.302790 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:33:21.302715 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26f57" podUID="fb66d280-cf24-4549-8fdb-89f16b6080c1" Oct 30 18:33:21.417307 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:21.417233 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:21.527832 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:21.527756 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:22.305410 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:22.305380 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dtppz" Oct 30 18:33:22.305560 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:33:22.305478 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dtppz" podUID="90488e60-cb70-4afa-a54b-6f333d345e84" Oct 30 18:33:23.302762 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:23.302732 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmdjg" Oct 30 18:33:23.303071 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:23.302732 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26f57" Oct 30 18:33:23.303071 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:33:23.302849 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zmdjg" podUID="e163025a-b19f-44e2-9058-b65f31bfbcc4" Oct 30 18:33:23.303376 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:33:23.303201 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26f57" podUID="fb66d280-cf24-4549-8fdb-89f16b6080c1" Oct 30 18:33:23.522818 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:23.522750 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-163.us-east-2.compute.internal" event="NodeReady" Oct 30 18:33:23.522938 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:23.522847 2575 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 18:33:23.623371 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:23.623346 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-w4clh"] Oct 30 18:33:23.625160 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:23.625127 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-w4clh" Oct 30 18:33:23.641597 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:23.641448 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 18:33:23.641786 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:23.641769 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 18:33:23.641839 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:23.641827 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-bqrtm\"" Oct 30 18:33:23.641877 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:23.641841 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 18:33:23.642357 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:23.642344 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 18:33:23.647810 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:23.647791 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-w4clh"] Oct 30 18:33:23.664661 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:23.664638 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-bz2mv"] Oct 30 18:33:23.666231 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:23.666217 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-vfn4g"] Oct 30 18:33:23.666351 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:23.666336 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bz2mv" Oct 30 18:33:23.668028 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:23.668012 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vfn4g" Oct 30 18:33:23.680894 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:23.680874 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vfn4g"] Oct 30 18:33:23.683254 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:23.683234 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bz2mv"] Oct 30 18:33:23.741377 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:23.741341 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 18:33:23.741531 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:23.741470 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-pg8bd\"" Oct 30 18:33:23.741531 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:23.741528 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 18:33:23.741616 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:23.741534 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 18:33:23.741616 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:23.741565 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-6twlp\"" Oct 30 18:33:23.741616 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:23.741576 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 18:33:23.741700 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:23.741623 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 18:33:23.771527 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:23.771502 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0a1badb7-be53-4b7f-933f-23e2599ab5ba-data-volume\") pod \"insights-runtime-extractor-w4clh\" (UID: \"0a1badb7-be53-4b7f-933f-23e2599ab5ba\") " pod="openshift-insights/insights-runtime-extractor-w4clh" Oct 30 18:33:23.771599 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:23.771550 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pf82\" (UniqueName: \"kubernetes.io/projected/0a1badb7-be53-4b7f-933f-23e2599ab5ba-kube-api-access-5pf82\") pod \"insights-runtime-extractor-w4clh\" (UID: \"0a1badb7-be53-4b7f-933f-23e2599ab5ba\") " pod="openshift-insights/insights-runtime-extractor-w4clh" Oct 30 18:33:23.771599 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:23.771586 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a1badb7-be53-4b7f-933f-23e2599ab5ba-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w4clh\" (UID: \"0a1badb7-be53-4b7f-933f-23e2599ab5ba\") " pod="openshift-insights/insights-runtime-extractor-w4clh" Oct 30 18:33:23.771659 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:23.771612 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6c1405e1-dd94-4151-9b4f-6392155a2023-config-volume\") pod \"dns-default-vfn4g\" (UID: \"6c1405e1-dd94-4151-9b4f-6392155a2023\") " pod="openshift-dns/dns-default-vfn4g" Oct 30 18:33:23.771690 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:23.771658 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c1405e1-dd94-4151-9b4f-6392155a2023-tmp-dir\") pod \"dns-default-vfn4g\" (UID: \"6c1405e1-dd94-4151-9b4f-6392155a2023\") " pod="openshift-dns/dns-default-vfn4g" Oct 30 18:33:23.771724 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:23.771713 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z8lm\" (UniqueName: \"kubernetes.io/projected/6c1405e1-dd94-4151-9b4f-6392155a2023-kube-api-access-6z8lm\") pod \"dns-default-vfn4g\" (UID: \"6c1405e1-dd94-4151-9b4f-6392155a2023\") " pod="openshift-dns/dns-default-vfn4g" Oct 30 18:33:23.771755 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:23.771740 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6c1405e1-dd94-4151-9b4f-6392155a2023-metrics-tls\") pod \"dns-default-vfn4g\" (UID: \"6c1405e1-dd94-4151-9b4f-6392155a2023\") " pod="openshift-dns/dns-default-vfn4g" Oct 30 18:33:23.771786 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:23.771770 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0a1badb7-be53-4b7f-933f-23e2599ab5ba-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w4clh\" (UID: \"0a1badb7-be53-4b7f-933f-23e2599ab5ba\") " pod="openshift-insights/insights-runtime-extractor-w4clh" Oct 30 18:33:23.771868 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:23.771855 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/91716cdd-99fb-4140-a52c-6d357ac74fb1-cert\") pod \"ingress-canary-bz2mv\" (UID: \"91716cdd-99fb-4140-a52c-6d357ac74fb1\") " pod="openshift-ingress-canary/ingress-canary-bz2mv" Oct 30 18:33:23.771901 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:23.771883 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbvzt\" (UniqueName: \"kubernetes.io/projected/91716cdd-99fb-4140-a52c-6d357ac74fb1-kube-api-access-bbvzt\") pod \"ingress-canary-bz2mv\" (UID: \"91716cdd-99fb-4140-a52c-6d357ac74fb1\") " pod="openshift-ingress-canary/ingress-canary-bz2mv" Oct 30 18:33:23.771933 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:23.771900 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0a1badb7-be53-4b7f-933f-23e2599ab5ba-crio-socket\") pod \"insights-runtime-extractor-w4clh\" (UID: \"0a1badb7-be53-4b7f-933f-23e2599ab5ba\") " pod="openshift-insights/insights-runtime-extractor-w4clh" Oct 30 18:33:23.872806 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:23.872774 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pf82\" (UniqueName: \"kubernetes.io/projected/0a1badb7-be53-4b7f-933f-23e2599ab5ba-kube-api-access-5pf82\") pod \"insights-runtime-extractor-w4clh\" (UID: \"0a1badb7-be53-4b7f-933f-23e2599ab5ba\") " pod="openshift-insights/insights-runtime-extractor-w4clh" Oct 30 18:33:23.872806 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:23.872811 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a1badb7-be53-4b7f-933f-23e2599ab5ba-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w4clh\" (UID: \"0a1badb7-be53-4b7f-933f-23e2599ab5ba\") " pod="openshift-insights/insights-runtime-extractor-w4clh" Oct 30 18:33:23.872987 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:23.872827 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6c1405e1-dd94-4151-9b4f-6392155a2023-config-volume\") pod \"dns-default-vfn4g\" (UID: \"6c1405e1-dd94-4151-9b4f-6392155a2023\") " pod="openshift-dns/dns-default-vfn4g" Oct 30 18:33:23.872987 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:23.872848 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c1405e1-dd94-4151-9b4f-6392155a2023-tmp-dir\") pod \"dns-default-vfn4g\" (UID: \"6c1405e1-dd94-4151-9b4f-6392155a2023\") " pod="openshift-dns/dns-default-vfn4g" Oct 30 18:33:23.873066 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:23.872978 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z8lm\" (UniqueName: \"kubernetes.io/projected/6c1405e1-dd94-4151-9b4f-6392155a2023-kube-api-access-6z8lm\") pod \"dns-default-vfn4g\" (UID: \"6c1405e1-dd94-4151-9b4f-6392155a2023\") " pod="openshift-dns/dns-default-vfn4g" Oct 30 18:33:23.873066 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:23.873022 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6c1405e1-dd94-4151-9b4f-6392155a2023-metrics-tls\") pod \"dns-default-vfn4g\" (UID: \"6c1405e1-dd94-4151-9b4f-6392155a2023\") " pod="openshift-dns/dns-default-vfn4g" Oct 30 18:33:23.873172 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:23.873066 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0a1badb7-be53-4b7f-933f-23e2599ab5ba-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w4clh\" (UID: \"0a1badb7-be53-4b7f-933f-23e2599ab5ba\") " pod="openshift-insights/insights-runtime-extractor-w4clh" Oct 30 18:33:23.873172 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:23.873113 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/91716cdd-99fb-4140-a52c-6d357ac74fb1-cert\") pod \"ingress-canary-bz2mv\" (UID: \"91716cdd-99fb-4140-a52c-6d357ac74fb1\") " pod="openshift-ingress-canary/ingress-canary-bz2mv" Oct 30 18:33:23.873172 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:23.873162 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbvzt\" (UniqueName: \"kubernetes.io/projected/91716cdd-99fb-4140-a52c-6d357ac74fb1-kube-api-access-bbvzt\") pod \"ingress-canary-bz2mv\" (UID: \"91716cdd-99fb-4140-a52c-6d357ac74fb1\") " pod="openshift-ingress-canary/ingress-canary-bz2mv" Oct 30 18:33:23.873330 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:23.873250 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c1405e1-dd94-4151-9b4f-6392155a2023-tmp-dir\") pod \"dns-default-vfn4g\" (UID: \"6c1405e1-dd94-4151-9b4f-6392155a2023\") " pod="openshift-dns/dns-default-vfn4g" Oct 30 18:33:23.873384 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:23.873370 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0a1badb7-be53-4b7f-933f-23e2599ab5ba-crio-socket\") pod \"insights-runtime-extractor-w4clh\" (UID: \"0a1badb7-be53-4b7f-933f-23e2599ab5ba\") " pod="openshift-insights/insights-runtime-extractor-w4clh" Oct 30 18:33:23.873440 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:23.873401 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0a1badb7-be53-4b7f-933f-23e2599ab5ba-data-volume\") pod \"insights-runtime-extractor-w4clh\" (UID: \"0a1badb7-be53-4b7f-933f-23e2599ab5ba\") " pod="openshift-insights/insights-runtime-extractor-w4clh" Oct 30 18:33:23.873606 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:23.873580 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6c1405e1-dd94-4151-9b4f-6392155a2023-config-volume\") pod \"dns-default-vfn4g\" (UID: \"6c1405e1-dd94-4151-9b4f-6392155a2023\") " pod="openshift-dns/dns-default-vfn4g" Oct 30 18:33:23.873746 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:23.873621 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0a1badb7-be53-4b7f-933f-23e2599ab5ba-crio-socket\") pod \"insights-runtime-extractor-w4clh\" (UID: \"0a1badb7-be53-4b7f-933f-23e2599ab5ba\") " pod="openshift-insights/insights-runtime-extractor-w4clh" Oct 30 18:33:23.873811 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:23.873692 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0a1badb7-be53-4b7f-933f-23e2599ab5ba-data-volume\") pod \"insights-runtime-extractor-w4clh\" (UID: \"0a1badb7-be53-4b7f-933f-23e2599ab5ba\") " pod="openshift-insights/insights-runtime-extractor-w4clh" Oct 30 18:33:23.873811 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:23.873694 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0a1badb7-be53-4b7f-933f-23e2599ab5ba-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w4clh\" (UID: \"0a1badb7-be53-4b7f-933f-23e2599ab5ba\") " pod="openshift-insights/insights-runtime-extractor-w4clh" Oct 30 18:33:23.876959 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:23.876928 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a1badb7-be53-4b7f-933f-23e2599ab5ba-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w4clh\" (UID: \"0a1badb7-be53-4b7f-933f-23e2599ab5ba\") " pod="openshift-insights/insights-runtime-extractor-w4clh" Oct 30 18:33:23.877040 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:23.877016 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6c1405e1-dd94-4151-9b4f-6392155a2023-metrics-tls\") pod \"dns-default-vfn4g\" (UID: \"6c1405e1-dd94-4151-9b4f-6392155a2023\") " pod="openshift-dns/dns-default-vfn4g" Oct 30 18:33:23.877096 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:23.877079 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/91716cdd-99fb-4140-a52c-6d357ac74fb1-cert\") pod \"ingress-canary-bz2mv\" (UID: \"91716cdd-99fb-4140-a52c-6d357ac74fb1\") " pod="openshift-ingress-canary/ingress-canary-bz2mv" Oct 30 18:33:23.885882 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:23.885857 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z8lm\" (UniqueName: \"kubernetes.io/projected/6c1405e1-dd94-4151-9b4f-6392155a2023-kube-api-access-6z8lm\") pod \"dns-default-vfn4g\" (UID: \"6c1405e1-dd94-4151-9b4f-6392155a2023\") " pod="openshift-dns/dns-default-vfn4g" Oct 30 18:33:23.885979 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:23.885952 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pf82\" (UniqueName: \"kubernetes.io/projected/0a1badb7-be53-4b7f-933f-23e2599ab5ba-kube-api-access-5pf82\") pod \"insights-runtime-extractor-w4clh\" (UID: \"0a1badb7-be53-4b7f-933f-23e2599ab5ba\") " pod="openshift-insights/insights-runtime-extractor-w4clh" Oct 30 18:33:23.886101 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:23.886079 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbvzt\" (UniqueName: \"kubernetes.io/projected/91716cdd-99fb-4140-a52c-6d357ac74fb1-kube-api-access-bbvzt\") pod \"ingress-canary-bz2mv\" (UID: \"91716cdd-99fb-4140-a52c-6d357ac74fb1\") " pod="openshift-ingress-canary/ingress-canary-bz2mv" Oct 30 18:33:23.937063 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:23.937039 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-w4clh" Oct 30 18:33:23.976086 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:23.976053 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bz2mv" Oct 30 18:33:23.982198 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:23.982162 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vfn4g" Oct 30 18:33:24.204083 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:24.204014 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:24.223320 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:24.223285 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-w4clh"] Oct 30 18:33:24.226659 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:33:24.226637 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a1badb7_be53_4b7f_933f_23e2599ab5ba.slice/crio-2c0ae33ab021d9f5442b117896d2dab11ba19c65301da78cb6bad9ac5d0117dc WatchSource:0}: Error finding container 2c0ae33ab021d9f5442b117896d2dab11ba19c65301da78cb6bad9ac5d0117dc: Status 404 returned error can't find the container with id 2c0ae33ab021d9f5442b117896d2dab11ba19c65301da78cb6bad9ac5d0117dc Oct 30 18:33:24.255932 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:24.255903 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bz2mv"] Oct 30 18:33:24.259780 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:33:24.259752 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91716cdd_99fb_4140_a52c_6d357ac74fb1.slice/crio-a3060082255cf6b51651149218af4bdbba6f1b0033acf76294ab9f552e69ec40 WatchSource:0}: Error finding container a3060082255cf6b51651149218af4bdbba6f1b0033acf76294ab9f552e69ec40: Status 404 returned error can't find the container with id a3060082255cf6b51651149218af4bdbba6f1b0033acf76294ab9f552e69ec40 Oct 30 18:33:24.263060 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:24.263034 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vfn4g"] Oct 30 18:33:24.268285 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:33:24.268061 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c1405e1_dd94_4151_9b4f_6392155a2023.slice/crio-095a58f8a000a16f80609d89f8adf94a3432280b70698e057daf05d423517c6f WatchSource:0}: Error finding container 095a58f8a000a16f80609d89f8adf94a3432280b70698e057daf05d423517c6f: Status 404 returned error can't find the container with id 095a58f8a000a16f80609d89f8adf94a3432280b70698e057daf05d423517c6f Oct 30 18:33:24.303193 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:24.303136 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dtppz" Oct 30 18:33:24.307772 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:24.307748 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-z87v4\"" Oct 30 18:33:24.307872 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:24.307840 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 18:33:24.321921 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:24.321889 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:24.427285 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:24.427252 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:24.539915 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:24.539806 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bz2mv" event={"ID":"91716cdd-99fb-4140-a52c-6d357ac74fb1","Type":"ContainerStarted","Data":"a3060082255cf6b51651149218af4bdbba6f1b0033acf76294ab9f552e69ec40"} Oct 30 18:33:24.544058 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:24.544025 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vfn4g" event={"ID":"6c1405e1-dd94-4151-9b4f-6392155a2023","Type":"ContainerStarted","Data":"095a58f8a000a16f80609d89f8adf94a3432280b70698e057daf05d423517c6f"} Oct 30 18:33:24.545382 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:24.545363 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w4clh" event={"ID":"0a1badb7-be53-4b7f-933f-23e2599ab5ba","Type":"ContainerStarted","Data":"8c55fbe17621fdf4f255a8930c508f684986814aae44d15394dde921f12f343f"} Oct 30 18:33:24.545459 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:24.545402 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w4clh" event={"ID":"0a1badb7-be53-4b7f-933f-23e2599ab5ba","Type":"ContainerStarted","Data":"2c0ae33ab021d9f5442b117896d2dab11ba19c65301da78cb6bad9ac5d0117dc"} Oct 30 18:33:24.591502 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:24.591474 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:25.090780 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:25.090743 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:25.194769 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:25.194663 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:25.303035 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:25.303006 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmdjg" Oct 30 18:33:25.303235 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:25.303047 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26f57" Oct 30 18:33:25.307948 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:25.307911 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 18:33:25.308075 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:25.307983 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 18:33:25.309768 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:25.309723 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-9qttr\"" Oct 30 18:33:25.309768 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:25.309738 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-fgz4w\"" Oct 30 18:33:25.309935 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:25.309811 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 18:33:25.550643 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:25.550553 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w4clh" event={"ID":"0a1badb7-be53-4b7f-933f-23e2599ab5ba","Type":"ContainerStarted","Data":"75ccc935621776c0b2ba5f2f3cf4c74418b83d3de2ce2f8e2e5069c2310c1ea2"} Oct 30 18:33:25.680454 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:25.680394 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:25.787008 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:25.786969 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:26.555811 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:26.555774 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vfn4g" event={"ID":"6c1405e1-dd94-4151-9b4f-6392155a2023","Type":"ContainerStarted","Data":"726c19ea4b0b8d41ceff91951ecfa3d5f9028a1b80ecb30e4ea59db7d709e6e3"} Oct 30 18:33:26.556309 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:26.555839 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vfn4g" event={"ID":"6c1405e1-dd94-4151-9b4f-6392155a2023","Type":"ContainerStarted","Data":"7db7615b095519bf741bbf81f7e08e4a80753d56a8080c44a505db55d58e15b6"} Oct 30 18:33:26.556309 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:26.555947 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-vfn4g" Oct 30 18:33:26.588403 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:26.588334 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-vfn4g" podStartSLOduration=1.754705971 podStartE2EDuration="3.588313531s" totalImagesPullingTime="1.83360756s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:33:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:33:26.587345674 +0000 UTC m=+44.806337121" watchObservedRunningTime="2026-10-30 18:33:26.588313531 +0000 UTC m=+44.807304955" Oct 30 18:33:26.639946 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:26.639906 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:26.795517 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:26.795479 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:26.808065 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:26.808003 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bk5br_05cad8b6-56f5-4bd1-b12d-b2ae456b7027/dns-node-resolver/0.log" Oct 30 18:33:26.855070 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:26.855040 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-8698b"] Oct 30 18:33:26.857128 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:26.857098 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8698b" Oct 30 18:33:26.876165 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:26.876123 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 30 18:33:26.876260 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:26.876174 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-rbnh2\"" Oct 30 18:33:26.879261 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:26.879244 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 18:33:26.879599 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:26.879581 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 18:33:26.880314 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:26.880296 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 18:33:26.886624 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:26.886603 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-8698b"] Oct 30 18:33:26.889100 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:26.889081 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-7gk7g"] Oct 30 18:33:26.889268 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:26.889252 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 30 18:33:26.892779 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:26.892756 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7gk7g" Oct 30 18:33:26.895684 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:26.895666 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-hh862"] Oct 30 18:33:26.897531 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:26.897515 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hh862" Oct 30 18:33:26.908697 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:26.908650 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 30 18:33:26.908697 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:26.908653 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-vsdjx\"" Oct 30 18:33:26.908857 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:26.908713 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-hh248\"" Oct 30 18:33:26.908857 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:26.908715 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 30 18:33:26.915359 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:26.915333 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-7gk7g"] Oct 30 18:33:26.976122 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:26.976091 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 18:33:26.977005 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:26.976981 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 18:33:26.978354 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:26.978323 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 18:33:26.978459 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:26.978408 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 30 18:33:26.995640 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:26.995614 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:26.998606 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:26.998580 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/433ed181-b45e-45c0-a593-9cb0b6c09348-metrics-client-ca\") pod \"node-exporter-hh862\" (UID: \"433ed181-b45e-45c0-a593-9cb0b6c09348\") " pod="openshift-monitoring/node-exporter-hh862" Oct 30 18:33:26.998714 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:26.998622 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/312dae04-dfda-41da-90ca-36b3b44a12ea-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-8698b\" (UID: \"312dae04-dfda-41da-90ca-36b3b44a12ea\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8698b" Oct 30 18:33:26.998714 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:26.998649 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/4f8841c0-4cf3-45d9-9301-957d4fffe1ce-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-7gk7g\" (UID: \"4f8841c0-4cf3-45d9-9301-957d4fffe1ce\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7gk7g" Oct 30 18:33:26.998714 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:26.998667 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/312dae04-dfda-41da-90ca-36b3b44a12ea-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-8698b\" (UID: \"312dae04-dfda-41da-90ca-36b3b44a12ea\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8698b" Oct 30 18:33:26.998714 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:26.998712 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/4f8841c0-4cf3-45d9-9301-957d4fffe1ce-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-7gk7g\" (UID: \"4f8841c0-4cf3-45d9-9301-957d4fffe1ce\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7gk7g" Oct 30 18:33:26.998904 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:26.998728 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4f8841c0-4cf3-45d9-9301-957d4fffe1ce-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-7gk7g\" (UID: \"4f8841c0-4cf3-45d9-9301-957d4fffe1ce\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7gk7g" Oct 30 18:33:26.998904 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:26.998744 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/433ed181-b45e-45c0-a593-9cb0b6c09348-node-exporter-accelerators-collector-config\") pod \"node-exporter-hh862\" (UID: \"433ed181-b45e-45c0-a593-9cb0b6c09348\") " pod="openshift-monitoring/node-exporter-hh862" Oct 30 18:33:26.998904 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:26.998774 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8422\" (UniqueName: \"kubernetes.io/projected/312dae04-dfda-41da-90ca-36b3b44a12ea-kube-api-access-z8422\") pod \"openshift-state-metrics-7bd7489957-8698b\" (UID: \"312dae04-dfda-41da-90ca-36b3b44a12ea\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8698b" Oct 30 18:33:26.998904 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:26.998800 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9nwr\" (UniqueName: \"kubernetes.io/projected/4f8841c0-4cf3-45d9-9301-957d4fffe1ce-kube-api-access-x9nwr\") pod \"kube-state-metrics-5dc8d7d89b-7gk7g\" (UID: \"4f8841c0-4cf3-45d9-9301-957d4fffe1ce\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7gk7g" Oct 30 18:33:26.998904 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:26.998821 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/433ed181-b45e-45c0-a593-9cb0b6c09348-node-exporter-tls\") pod \"node-exporter-hh862\" (UID: \"433ed181-b45e-45c0-a593-9cb0b6c09348\") " pod="openshift-monitoring/node-exporter-hh862" Oct 30 18:33:26.998904 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:26.998847 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4f8841c0-4cf3-45d9-9301-957d4fffe1ce-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-7gk7g\" (UID: \"4f8841c0-4cf3-45d9-9301-957d4fffe1ce\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7gk7g" Oct 30 18:33:26.998904 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:26.998879 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/433ed181-b45e-45c0-a593-9cb0b6c09348-node-exporter-textfile\") pod \"node-exporter-hh862\" (UID: \"433ed181-b45e-45c0-a593-9cb0b6c09348\") " pod="openshift-monitoring/node-exporter-hh862" Oct 30 18:33:26.999093 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:26.998923 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4f8841c0-4cf3-45d9-9301-957d4fffe1ce-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-7gk7g\" (UID: \"4f8841c0-4cf3-45d9-9301-957d4fffe1ce\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7gk7g" Oct 30 18:33:26.999093 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:26.998947 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/312dae04-dfda-41da-90ca-36b3b44a12ea-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-8698b\" (UID: \"312dae04-dfda-41da-90ca-36b3b44a12ea\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8698b" Oct 30 18:33:26.999093 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:26.998963 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/433ed181-b45e-45c0-a593-9cb0b6c09348-sys\") pod \"node-exporter-hh862\" (UID: \"433ed181-b45e-45c0-a593-9cb0b6c09348\") " pod="openshift-monitoring/node-exporter-hh862" Oct 30 18:33:26.999093 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:26.998976 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/433ed181-b45e-45c0-a593-9cb0b6c09348-root\") pod \"node-exporter-hh862\" (UID: \"433ed181-b45e-45c0-a593-9cb0b6c09348\") " pod="openshift-monitoring/node-exporter-hh862" Oct 30 18:33:26.999093 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:26.998991 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/433ed181-b45e-45c0-a593-9cb0b6c09348-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hh862\" (UID: \"433ed181-b45e-45c0-a593-9cb0b6c09348\") " pod="openshift-monitoring/node-exporter-hh862" Oct 30 18:33:26.999093 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:26.999020 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g52xm\" (UniqueName: \"kubernetes.io/projected/433ed181-b45e-45c0-a593-9cb0b6c09348-kube-api-access-g52xm\") pod \"node-exporter-hh862\" (UID: \"433ed181-b45e-45c0-a593-9cb0b6c09348\") " pod="openshift-monitoring/node-exporter-hh862" Oct 30 18:33:26.999093 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:26.999041 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/433ed181-b45e-45c0-a593-9cb0b6c09348-node-exporter-wtmp\") pod \"node-exporter-hh862\" (UID: \"433ed181-b45e-45c0-a593-9cb0b6c09348\") " pod="openshift-monitoring/node-exporter-hh862" Oct 30 18:33:27.099652 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:27.099627 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/433ed181-b45e-45c0-a593-9cb0b6c09348-node-exporter-wtmp\") pod \"node-exporter-hh862\" (UID: \"433ed181-b45e-45c0-a593-9cb0b6c09348\") " pod="openshift-monitoring/node-exporter-hh862" Oct 30 18:33:27.099767 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:27.099665 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/433ed181-b45e-45c0-a593-9cb0b6c09348-metrics-client-ca\") pod \"node-exporter-hh862\" (UID: \"433ed181-b45e-45c0-a593-9cb0b6c09348\") " pod="openshift-monitoring/node-exporter-hh862" Oct 30 18:33:27.099767 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:27.099695 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/312dae04-dfda-41da-90ca-36b3b44a12ea-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-8698b\" (UID: \"312dae04-dfda-41da-90ca-36b3b44a12ea\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8698b" Oct 30 18:33:27.099767 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:27.099723 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/4f8841c0-4cf3-45d9-9301-957d4fffe1ce-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-7gk7g\" (UID: \"4f8841c0-4cf3-45d9-9301-957d4fffe1ce\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7gk7g" Oct 30 18:33:27.099767 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:27.099750 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/312dae04-dfda-41da-90ca-36b3b44a12ea-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-8698b\" (UID: \"312dae04-dfda-41da-90ca-36b3b44a12ea\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8698b" Oct 30 18:33:27.099926 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:27.099792 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/433ed181-b45e-45c0-a593-9cb0b6c09348-node-exporter-wtmp\") pod \"node-exporter-hh862\" (UID: \"433ed181-b45e-45c0-a593-9cb0b6c09348\") " pod="openshift-monitoring/node-exporter-hh862" Oct 30 18:33:27.099926 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:27.099847 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/4f8841c0-4cf3-45d9-9301-957d4fffe1ce-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-7gk7g\" (UID: \"4f8841c0-4cf3-45d9-9301-957d4fffe1ce\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7gk7g" Oct 30 18:33:27.099926 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:27.099884 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4f8841c0-4cf3-45d9-9301-957d4fffe1ce-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-7gk7g\" (UID: \"4f8841c0-4cf3-45d9-9301-957d4fffe1ce\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7gk7g" Oct 30 18:33:27.099926 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:27.099912 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/433ed181-b45e-45c0-a593-9cb0b6c09348-node-exporter-accelerators-collector-config\") pod \"node-exporter-hh862\" (UID: \"433ed181-b45e-45c0-a593-9cb0b6c09348\") " pod="openshift-monitoring/node-exporter-hh862" Oct 30 18:33:27.100100 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:27.099943 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8422\" (UniqueName: \"kubernetes.io/projected/312dae04-dfda-41da-90ca-36b3b44a12ea-kube-api-access-z8422\") pod \"openshift-state-metrics-7bd7489957-8698b\" (UID: \"312dae04-dfda-41da-90ca-36b3b44a12ea\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8698b" Oct 30 18:33:27.100100 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:27.099969 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9nwr\" (UniqueName: \"kubernetes.io/projected/4f8841c0-4cf3-45d9-9301-957d4fffe1ce-kube-api-access-x9nwr\") pod \"kube-state-metrics-5dc8d7d89b-7gk7g\" (UID: \"4f8841c0-4cf3-45d9-9301-957d4fffe1ce\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7gk7g" Oct 30 18:33:27.100100 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:27.099994 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/433ed181-b45e-45c0-a593-9cb0b6c09348-node-exporter-tls\") pod \"node-exporter-hh862\" (UID: \"433ed181-b45e-45c0-a593-9cb0b6c09348\") " pod="openshift-monitoring/node-exporter-hh862" Oct 30 18:33:27.100100 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:27.100024 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4f8841c0-4cf3-45d9-9301-957d4fffe1ce-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-7gk7g\" (UID: \"4f8841c0-4cf3-45d9-9301-957d4fffe1ce\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7gk7g" Oct 30 18:33:27.100100 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:27.100058 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/433ed181-b45e-45c0-a593-9cb0b6c09348-node-exporter-textfile\") pod \"node-exporter-hh862\" (UID: \"433ed181-b45e-45c0-a593-9cb0b6c09348\") " pod="openshift-monitoring/node-exporter-hh862" Oct 30 18:33:27.100344 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:27.100104 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4f8841c0-4cf3-45d9-9301-957d4fffe1ce-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-7gk7g\" (UID: \"4f8841c0-4cf3-45d9-9301-957d4fffe1ce\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7gk7g" Oct 30 18:33:27.100344 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:27.100158 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/312dae04-dfda-41da-90ca-36b3b44a12ea-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-8698b\" (UID: \"312dae04-dfda-41da-90ca-36b3b44a12ea\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8698b" Oct 30 18:33:27.100344 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:27.100187 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/433ed181-b45e-45c0-a593-9cb0b6c09348-sys\") pod \"node-exporter-hh862\" (UID: \"433ed181-b45e-45c0-a593-9cb0b6c09348\") " pod="openshift-monitoring/node-exporter-hh862" Oct 30 18:33:27.100344 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:27.100210 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/433ed181-b45e-45c0-a593-9cb0b6c09348-root\") pod \"node-exporter-hh862\" (UID: \"433ed181-b45e-45c0-a593-9cb0b6c09348\") " pod="openshift-monitoring/node-exporter-hh862" Oct 30 18:33:27.100344 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:27.100234 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/433ed181-b45e-45c0-a593-9cb0b6c09348-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hh862\" (UID: \"433ed181-b45e-45c0-a593-9cb0b6c09348\") " pod="openshift-monitoring/node-exporter-hh862" Oct 30 18:33:27.100344 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:27.100247 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/4f8841c0-4cf3-45d9-9301-957d4fffe1ce-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-7gk7g\" (UID: \"4f8841c0-4cf3-45d9-9301-957d4fffe1ce\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7gk7g" Oct 30 18:33:27.100344 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:27.100303 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/433ed181-b45e-45c0-a593-9cb0b6c09348-metrics-client-ca\") pod \"node-exporter-hh862\" (UID: \"433ed181-b45e-45c0-a593-9cb0b6c09348\") " pod="openshift-monitoring/node-exporter-hh862" Oct 30 18:33:27.100344 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:27.100304 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g52xm\" (UniqueName: \"kubernetes.io/projected/433ed181-b45e-45c0-a593-9cb0b6c09348-kube-api-access-g52xm\") pod \"node-exporter-hh862\" (UID: \"433ed181-b45e-45c0-a593-9cb0b6c09348\") " pod="openshift-monitoring/node-exporter-hh862" Oct 30 18:33:27.100744 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:27.100546 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/312dae04-dfda-41da-90ca-36b3b44a12ea-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-8698b\" (UID: \"312dae04-dfda-41da-90ca-36b3b44a12ea\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8698b" Oct 30 18:33:27.100744 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:27.100665 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/433ed181-b45e-45c0-a593-9cb0b6c09348-node-exporter-textfile\") pod \"node-exporter-hh862\" (UID: \"433ed181-b45e-45c0-a593-9cb0b6c09348\") " pod="openshift-monitoring/node-exporter-hh862" Oct 30 18:33:27.100744 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:27.100684 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/4f8841c0-4cf3-45d9-9301-957d4fffe1ce-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-7gk7g\" (UID: \"4f8841c0-4cf3-45d9-9301-957d4fffe1ce\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7gk7g" Oct 30 18:33:27.100744 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:27.100730 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/433ed181-b45e-45c0-a593-9cb0b6c09348-sys\") pod \"node-exporter-hh862\" (UID: \"433ed181-b45e-45c0-a593-9cb0b6c09348\") " pod="openshift-monitoring/node-exporter-hh862" Oct 30 18:33:27.100968 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:27.100781 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/433ed181-b45e-45c0-a593-9cb0b6c09348-root\") pod \"node-exporter-hh862\" (UID: \"433ed181-b45e-45c0-a593-9cb0b6c09348\") " pod="openshift-monitoring/node-exporter-hh862" Oct 30 18:33:27.101037 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:27.100979 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4f8841c0-4cf3-45d9-9301-957d4fffe1ce-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-7gk7g\" (UID: \"4f8841c0-4cf3-45d9-9301-957d4fffe1ce\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7gk7g" Oct 30 18:33:27.101358 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:27.101330 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/433ed181-b45e-45c0-a593-9cb0b6c09348-node-exporter-accelerators-collector-config\") pod \"node-exporter-hh862\" (UID: \"433ed181-b45e-45c0-a593-9cb0b6c09348\") " pod="openshift-monitoring/node-exporter-hh862" Oct 30 18:33:27.103509 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:27.103484 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/433ed181-b45e-45c0-a593-9cb0b6c09348-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hh862\" (UID: \"433ed181-b45e-45c0-a593-9cb0b6c09348\") " pod="openshift-monitoring/node-exporter-hh862" Oct 30 18:33:27.103686 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:27.103661 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4f8841c0-4cf3-45d9-9301-957d4fffe1ce-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-7gk7g\" (UID: \"4f8841c0-4cf3-45d9-9301-957d4fffe1ce\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7gk7g" Oct 30 18:33:27.103752 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:27.103735 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/312dae04-dfda-41da-90ca-36b3b44a12ea-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-8698b\" (UID: \"312dae04-dfda-41da-90ca-36b3b44a12ea\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8698b" Oct 30 18:33:27.103880 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:27.103856 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/433ed181-b45e-45c0-a593-9cb0b6c09348-node-exporter-tls\") pod \"node-exporter-hh862\" (UID: \"433ed181-b45e-45c0-a593-9cb0b6c09348\") " pod="openshift-monitoring/node-exporter-hh862" Oct 30 18:33:27.103938 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:27.103902 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/312dae04-dfda-41da-90ca-36b3b44a12ea-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-8698b\" (UID: \"312dae04-dfda-41da-90ca-36b3b44a12ea\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8698b" Oct 30 18:33:27.103938 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:27.103920 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4f8841c0-4cf3-45d9-9301-957d4fffe1ce-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-7gk7g\" (UID: \"4f8841c0-4cf3-45d9-9301-957d4fffe1ce\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7gk7g" Oct 30 18:33:27.117509 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:27.117464 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8422\" (UniqueName: \"kubernetes.io/projected/312dae04-dfda-41da-90ca-36b3b44a12ea-kube-api-access-z8422\") pod \"openshift-state-metrics-7bd7489957-8698b\" (UID: \"312dae04-dfda-41da-90ca-36b3b44a12ea\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8698b" Oct 30 18:33:27.117675 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:27.117658 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g52xm\" (UniqueName: \"kubernetes.io/projected/433ed181-b45e-45c0-a593-9cb0b6c09348-kube-api-access-g52xm\") pod \"node-exporter-hh862\" (UID: \"433ed181-b45e-45c0-a593-9cb0b6c09348\") " pod="openshift-monitoring/node-exporter-hh862" Oct 30 18:33:27.117829 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:27.117807 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9nwr\" (UniqueName: \"kubernetes.io/projected/4f8841c0-4cf3-45d9-9301-957d4fffe1ce-kube-api-access-x9nwr\") pod \"kube-state-metrics-5dc8d7d89b-7gk7g\" (UID: \"4f8841c0-4cf3-45d9-9301-957d4fffe1ce\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7gk7g" Oct 30 18:33:27.119102 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:27.119078 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:27.170337 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:27.170314 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8698b" Oct 30 18:33:27.209461 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:27.209430 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7gk7g" Oct 30 18:33:27.225425 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:27.225044 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hh862" Oct 30 18:33:27.227355 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:27.227249 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:27.239263 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:33:27.239230 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod433ed181_b45e_45c0_a593_9cb0b6c09348.slice/crio-cf60a7a7a606fdc05b761a1d4e2408b15e4ec237e5a5c22d1b6a49bde2a24ad5 WatchSource:0}: Error finding container cf60a7a7a606fdc05b761a1d4e2408b15e4ec237e5a5c22d1b6a49bde2a24ad5: Status 404 returned error can't find the container with id cf60a7a7a606fdc05b761a1d4e2408b15e4ec237e5a5c22d1b6a49bde2a24ad5 Oct 30 18:33:27.343788 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:27.343712 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:27.442493 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:27.442474 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-8698b"] Oct 30 18:33:27.444259 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:33:27.444238 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod312dae04_dfda_41da_90ca_36b3b44a12ea.slice/crio-683740f7ded55252e4418cbf3eef7b202800bc07ad67aee5e15472576a901d63 WatchSource:0}: Error finding container 683740f7ded55252e4418cbf3eef7b202800bc07ad67aee5e15472576a901d63: Status 404 returned error can't find the container with id 683740f7ded55252e4418cbf3eef7b202800bc07ad67aee5e15472576a901d63 Oct 30 18:33:27.508385 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:27.508363 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-7gk7g"] Oct 30 18:33:27.514660 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:33:27.514630 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f8841c0_4cf3_45d9_9301_957d4fffe1ce.slice/crio-6f84cca17983b8b70b45b2bcdc32e64617a0f4bb6889b0c8fa47567b6a7149eb WatchSource:0}: Error finding container 6f84cca17983b8b70b45b2bcdc32e64617a0f4bb6889b0c8fa47567b6a7149eb: Status 404 returned error can't find the container with id 6f84cca17983b8b70b45b2bcdc32e64617a0f4bb6889b0c8fa47567b6a7149eb Oct 30 18:33:27.561880 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:27.561850 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-w4clh" event={"ID":"0a1badb7-be53-4b7f-933f-23e2599ab5ba","Type":"ContainerStarted","Data":"df199ac284432671c1a5cfbb4b68abf646cd1174fbfe55888b17c9a1631b7e28"} Oct 30 18:33:27.564326 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:27.564299 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bz2mv" event={"ID":"91716cdd-99fb-4140-a52c-6d357ac74fb1","Type":"ContainerStarted","Data":"9c7ff60a21f882ca42b0604f921823fd03c948fda4a5b1d9cc4adb4e3157cc23"} Oct 30 18:33:27.565476 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:27.565443 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7gk7g" event={"ID":"4f8841c0-4cf3-45d9-9301-957d4fffe1ce","Type":"ContainerStarted","Data":"6f84cca17983b8b70b45b2bcdc32e64617a0f4bb6889b0c8fa47567b6a7149eb"} Oct 30 18:33:27.567128 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:27.567098 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hh862" event={"ID":"433ed181-b45e-45c0-a593-9cb0b6c09348","Type":"ContainerStarted","Data":"cf60a7a7a606fdc05b761a1d4e2408b15e4ec237e5a5c22d1b6a49bde2a24ad5"} Oct 30 18:33:27.569040 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:27.569021 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8698b" event={"ID":"312dae04-dfda-41da-90ca-36b3b44a12ea","Type":"ContainerStarted","Data":"5f13e0891c9cd5d81ca093cd4bb99bb29c8bb519b81901dd11ed2e4c7bcbbf29"} Oct 30 18:33:27.569125 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:27.569062 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8698b" event={"ID":"312dae04-dfda-41da-90ca-36b3b44a12ea","Type":"ContainerStarted","Data":"333fac19757909768f614aa5caa3e1224dfcabb9bf14122f2282bd15a20959ca"} Oct 30 18:33:27.569125 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:27.569084 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8698b" event={"ID":"312dae04-dfda-41da-90ca-36b3b44a12ea","Type":"ContainerStarted","Data":"683740f7ded55252e4418cbf3eef7b202800bc07ad67aee5e15472576a901d63"} Oct 30 18:33:27.588038 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:27.588001 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-w4clh" podStartSLOduration=2.313396477 podStartE2EDuration="4.587988849s" totalImagesPullingTime="2.274592372s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:33:23 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:33:27.587043128 +0000 UTC m=+45.806034549" watchObservedRunningTime="2026-10-30 18:33:27.587988849 +0000 UTC m=+45.806980271" Oct 30 18:33:27.608334 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:27.608253 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-bz2mv" podStartSLOduration=1.81618505 podStartE2EDuration="4.60824409s" totalImagesPullingTime="2.79205904s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:33:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:33:27.607689928 +0000 UTC m=+45.826681351" watchObservedRunningTime="2026-10-30 18:33:27.60824409 +0000 UTC m=+45.827235514" Oct 30 18:33:27.667805 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:27.667775 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:27.774083 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:27.774051 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:27.884543 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:27.884467 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:27.889560 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:27.889524 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 18:33:27.891998 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:27.891979 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:33:27.896598 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:27.896574 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 18:33:27.897532 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:27.897500 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 18:33:27.901085 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:27.898425 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 18:33:27.901085 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:27.899438 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-h4dl6\"" Oct 30 18:33:27.901085 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:27.899494 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 18:33:27.901085 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:27.899500 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 18:33:27.901085 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:27.899608 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 18:33:27.901085 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:27.899714 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 18:33:27.901085 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:27.899736 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 18:33:27.901085 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:27.899852 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 18:33:27.912374 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:27.912351 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 18:33:28.001032 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:28.000997 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:28.007083 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:28.007060 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:33:28.007215 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:28.007102 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:33:28.007215 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:28.007126 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:33:28.007215 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:28.007204 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:33:28.007360 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:28.007258 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25trq\" (UniqueName: \"kubernetes.io/projected/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-kube-api-access-25trq\") pod \"alertmanager-main-0\" (UID: \"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:33:28.007360 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:28.007280 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:33:28.007360 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:28.007299 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:33:28.007474 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:28.007387 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:33:28.007474 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:28.007465 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:33:28.007564 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:28.007498 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-web-config\") pod \"alertmanager-main-0\" (UID: \"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:33:28.007597 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:28.007558 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:33:28.007644 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:28.007620 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-config-volume\") pod \"alertmanager-main-0\" (UID: \"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:33:28.007675 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:28.007651 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-config-out\") pod \"alertmanager-main-0\" (UID: \"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:33:28.108202 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:28.108162 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:33:28.108202 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:28.108219 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:33:28.108421 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:28.108245 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-web-config\") pod \"alertmanager-main-0\" (UID: \"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:33:28.108421 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:28.108270 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:33:28.108421 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:28.108316 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-config-volume\") pod \"alertmanager-main-0\" (UID: \"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:33:28.108421 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:28.108381 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-config-out\") pod \"alertmanager-main-0\" (UID: \"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:33:28.108421 ip-10-0-140-163 kubenswrapper[2575]: E1030 18:33:28.108404 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-alertmanager-trusted-ca-bundle podName:7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d nodeName:}" failed. No retries permitted until 2026-10-30 18:33:28.608380764 +0000 UTC m=+46.827372192 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d") : configmap references non-existent config key: ca-bundle.crt Oct 30 18:33:28.109853 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:28.108688 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:33:28.109853 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:28.108740 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:33:28.109853 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:28.108774 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:33:28.109853 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:28.108806 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:33:28.109853 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:28.108851 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25trq\" (UniqueName: \"kubernetes.io/projected/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-kube-api-access-25trq\") pod \"alertmanager-main-0\" (UID: \"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:33:28.109853 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:28.108883 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:33:28.109853 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:28.108905 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:33:28.109853 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:28.109245 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:33:28.110517 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:28.109857 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:33:28.111756 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:28.111730 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:33:28.112845 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:28.112821 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:33:28.113931 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:28.113261 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-web-config\") pod \"alertmanager-main-0\" (UID: \"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:33:28.113931 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:28.113320 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:33:28.113931 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:28.113678 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:33:28.113931 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:28.113904 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:33:28.114122 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:28.114082 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-config-volume\") pod \"alertmanager-main-0\" (UID: \"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:33:28.114688 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:28.114668 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:33:28.116322 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:28.116293 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-config-out\") pod \"alertmanager-main-0\" (UID: \"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:33:28.122961 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:28.122938 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25trq\" (UniqueName: \"kubernetes.io/projected/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-kube-api-access-25trq\") pod \"alertmanager-main-0\" (UID: \"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:33:28.219634 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:28.219505 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:28.360072 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:28.359986 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:28.406699 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:28.406671 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qrdc6_b0f929fc-a9ea-4935-9fe7-770c5e7d4de5/node-ca/0.log" Oct 30 18:33:28.574220 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:28.574187 2575 generic.go:356] "Generic (PLEG): container finished" podID="433ed181-b45e-45c0-a593-9cb0b6c09348" containerID="00564a87b49da301f403b8a788c97f463906f216d56704076f1c53843f0fe6a8" exitCode=0 Oct 30 18:33:28.574539 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:28.574270 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hh862" event={"ID":"433ed181-b45e-45c0-a593-9cb0b6c09348","Type":"ContainerDied","Data":"00564a87b49da301f403b8a788c97f463906f216d56704076f1c53843f0fe6a8"} Oct 30 18:33:28.612427 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:28.612393 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:33:28.613303 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:28.613285 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:33:28.811715 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:28.811687 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:33:28.825995 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:28.825973 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7c6f76d6d-k2h7h"] Oct 30 18:33:28.829366 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:28.829343 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7c6f76d6d-k2h7h" Oct 30 18:33:28.830364 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:28.830338 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bz2mv_91716cdd-99fb-4140-a52c-6d357ac74fb1/serve-healthcheck-canary/0.log" Oct 30 18:33:28.839897 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:28.839875 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-7hx8m\"" Oct 30 18:33:28.857381 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:28.857348 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7c6f76d6d-k2h7h"] Oct 30 18:33:28.939308 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:28.939273 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 30 18:33:28.939308 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:28.939273 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 30 18:33:28.939475 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:28.939310 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 30 18:33:28.939475 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:28.939316 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 30 18:33:28.939475 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:28.939324 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 30 18:33:28.939475 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:28.939368 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-b062eu2aaotn4\"" Oct 30 18:33:29.014928 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:29.014897 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/158becf0-d6cf-4917-8329-4f64ae5a4ca4-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7c6f76d6d-k2h7h\" (UID: \"158becf0-d6cf-4917-8329-4f64ae5a4ca4\") " pod="openshift-monitoring/thanos-querier-7c6f76d6d-k2h7h" Oct 30 18:33:29.015027 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:29.014957 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/158becf0-d6cf-4917-8329-4f64ae5a4ca4-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7c6f76d6d-k2h7h\" (UID: \"158becf0-d6cf-4917-8329-4f64ae5a4ca4\") " pod="openshift-monitoring/thanos-querier-7c6f76d6d-k2h7h" Oct 30 18:33:29.015027 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:29.015009 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/158becf0-d6cf-4917-8329-4f64ae5a4ca4-secret-grpc-tls\") pod \"thanos-querier-7c6f76d6d-k2h7h\" (UID: \"158becf0-d6cf-4917-8329-4f64ae5a4ca4\") " pod="openshift-monitoring/thanos-querier-7c6f76d6d-k2h7h" Oct 30 18:33:29.015129 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:29.015040 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/158becf0-d6cf-4917-8329-4f64ae5a4ca4-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7c6f76d6d-k2h7h\" (UID: \"158becf0-d6cf-4917-8329-4f64ae5a4ca4\") " pod="openshift-monitoring/thanos-querier-7c6f76d6d-k2h7h" Oct 30 18:33:29.015282 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:29.015258 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/158becf0-d6cf-4917-8329-4f64ae5a4ca4-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7c6f76d6d-k2h7h\" (UID: \"158becf0-d6cf-4917-8329-4f64ae5a4ca4\") " pod="openshift-monitoring/thanos-querier-7c6f76d6d-k2h7h" Oct 30 18:33:29.015350 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:29.015300 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/158becf0-d6cf-4917-8329-4f64ae5a4ca4-secret-thanos-querier-tls\") pod \"thanos-querier-7c6f76d6d-k2h7h\" (UID: \"158becf0-d6cf-4917-8329-4f64ae5a4ca4\") " pod="openshift-monitoring/thanos-querier-7c6f76d6d-k2h7h" Oct 30 18:33:29.015350 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:29.015336 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/158becf0-d6cf-4917-8329-4f64ae5a4ca4-metrics-client-ca\") pod \"thanos-querier-7c6f76d6d-k2h7h\" (UID: \"158becf0-d6cf-4917-8329-4f64ae5a4ca4\") " pod="openshift-monitoring/thanos-querier-7c6f76d6d-k2h7h" Oct 30 18:33:29.015446 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:29.015393 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/158becf0-d6cf-4917-8329-4f64ae5a4ca4-tmp\") pod \"thanos-querier-7c6f76d6d-k2h7h\" (UID: \"158becf0-d6cf-4917-8329-4f64ae5a4ca4\") " pod="openshift-monitoring/thanos-querier-7c6f76d6d-k2h7h" Oct 30 18:33:29.015497 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:29.015466 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zndl\" (UniqueName: \"kubernetes.io/projected/158becf0-d6cf-4917-8329-4f64ae5a4ca4-kube-api-access-4zndl\") pod \"thanos-querier-7c6f76d6d-k2h7h\" (UID: \"158becf0-d6cf-4917-8329-4f64ae5a4ca4\") " pod="openshift-monitoring/thanos-querier-7c6f76d6d-k2h7h" Oct 30 18:33:29.113119 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:29.107864 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 18:33:29.117194 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:29.116464 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/158becf0-d6cf-4917-8329-4f64ae5a4ca4-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7c6f76d6d-k2h7h\" (UID: \"158becf0-d6cf-4917-8329-4f64ae5a4ca4\") " pod="openshift-monitoring/thanos-querier-7c6f76d6d-k2h7h" Oct 30 18:33:29.117194 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:29.116505 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/158becf0-d6cf-4917-8329-4f64ae5a4ca4-secret-grpc-tls\") pod \"thanos-querier-7c6f76d6d-k2h7h\" (UID: \"158becf0-d6cf-4917-8329-4f64ae5a4ca4\") " pod="openshift-monitoring/thanos-querier-7c6f76d6d-k2h7h" Oct 30 18:33:29.117194 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:29.116525 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/158becf0-d6cf-4917-8329-4f64ae5a4ca4-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7c6f76d6d-k2h7h\" (UID: \"158becf0-d6cf-4917-8329-4f64ae5a4ca4\") " pod="openshift-monitoring/thanos-querier-7c6f76d6d-k2h7h" Oct 30 18:33:29.117194 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:29.116567 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/158becf0-d6cf-4917-8329-4f64ae5a4ca4-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7c6f76d6d-k2h7h\" (UID: \"158becf0-d6cf-4917-8329-4f64ae5a4ca4\") " pod="openshift-monitoring/thanos-querier-7c6f76d6d-k2h7h" Oct 30 18:33:29.117194 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:29.116587 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/158becf0-d6cf-4917-8329-4f64ae5a4ca4-secret-thanos-querier-tls\") pod \"thanos-querier-7c6f76d6d-k2h7h\" (UID: \"158becf0-d6cf-4917-8329-4f64ae5a4ca4\") " pod="openshift-monitoring/thanos-querier-7c6f76d6d-k2h7h" Oct 30 18:33:29.117194 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:29.116610 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/158becf0-d6cf-4917-8329-4f64ae5a4ca4-metrics-client-ca\") pod \"thanos-querier-7c6f76d6d-k2h7h\" (UID: \"158becf0-d6cf-4917-8329-4f64ae5a4ca4\") " pod="openshift-monitoring/thanos-querier-7c6f76d6d-k2h7h" Oct 30 18:33:29.117194 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:29.116626 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/158becf0-d6cf-4917-8329-4f64ae5a4ca4-tmp\") pod \"thanos-querier-7c6f76d6d-k2h7h\" (UID: \"158becf0-d6cf-4917-8329-4f64ae5a4ca4\") " pod="openshift-monitoring/thanos-querier-7c6f76d6d-k2h7h" Oct 30 18:33:29.117194 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:29.116644 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zndl\" (UniqueName: \"kubernetes.io/projected/158becf0-d6cf-4917-8329-4f64ae5a4ca4-kube-api-access-4zndl\") pod \"thanos-querier-7c6f76d6d-k2h7h\" (UID: \"158becf0-d6cf-4917-8329-4f64ae5a4ca4\") " pod="openshift-monitoring/thanos-querier-7c6f76d6d-k2h7h" Oct 30 18:33:29.117194 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:29.116681 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/158becf0-d6cf-4917-8329-4f64ae5a4ca4-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7c6f76d6d-k2h7h\" (UID: \"158becf0-d6cf-4917-8329-4f64ae5a4ca4\") " pod="openshift-monitoring/thanos-querier-7c6f76d6d-k2h7h" Oct 30 18:33:29.119177 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:29.118074 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/158becf0-d6cf-4917-8329-4f64ae5a4ca4-metrics-client-ca\") pod \"thanos-querier-7c6f76d6d-k2h7h\" (UID: \"158becf0-d6cf-4917-8329-4f64ae5a4ca4\") " pod="openshift-monitoring/thanos-querier-7c6f76d6d-k2h7h" Oct 30 18:33:29.119177 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:29.118793 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/158becf0-d6cf-4917-8329-4f64ae5a4ca4-tmp\") pod \"thanos-querier-7c6f76d6d-k2h7h\" (UID: \"158becf0-d6cf-4917-8329-4f64ae5a4ca4\") " pod="openshift-monitoring/thanos-querier-7c6f76d6d-k2h7h" Oct 30 18:33:29.121159 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:29.121118 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/158becf0-d6cf-4917-8329-4f64ae5a4ca4-secret-grpc-tls\") pod \"thanos-querier-7c6f76d6d-k2h7h\" (UID: \"158becf0-d6cf-4917-8329-4f64ae5a4ca4\") " pod="openshift-monitoring/thanos-querier-7c6f76d6d-k2h7h" Oct 30 18:33:29.121616 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:29.121596 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/158becf0-d6cf-4917-8329-4f64ae5a4ca4-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7c6f76d6d-k2h7h\" (UID: \"158becf0-d6cf-4917-8329-4f64ae5a4ca4\") " pod="openshift-monitoring/thanos-querier-7c6f76d6d-k2h7h" Oct 30 18:33:29.122093 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:29.122050 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/158becf0-d6cf-4917-8329-4f64ae5a4ca4-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7c6f76d6d-k2h7h\" (UID: \"158becf0-d6cf-4917-8329-4f64ae5a4ca4\") " pod="openshift-monitoring/thanos-querier-7c6f76d6d-k2h7h" Oct 30 18:33:29.125084 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:29.124661 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/158becf0-d6cf-4917-8329-4f64ae5a4ca4-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7c6f76d6d-k2h7h\" (UID: \"158becf0-d6cf-4917-8329-4f64ae5a4ca4\") " pod="openshift-monitoring/thanos-querier-7c6f76d6d-k2h7h" Oct 30 18:33:29.125084 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:29.124941 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/158becf0-d6cf-4917-8329-4f64ae5a4ca4-secret-thanos-querier-tls\") pod \"thanos-querier-7c6f76d6d-k2h7h\" (UID: \"158becf0-d6cf-4917-8329-4f64ae5a4ca4\") " pod="openshift-monitoring/thanos-querier-7c6f76d6d-k2h7h" Oct 30 18:33:29.126666 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:29.126611 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/158becf0-d6cf-4917-8329-4f64ae5a4ca4-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7c6f76d6d-k2h7h\" (UID: \"158becf0-d6cf-4917-8329-4f64ae5a4ca4\") " pod="openshift-monitoring/thanos-querier-7c6f76d6d-k2h7h" Oct 30 18:33:29.138020 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:29.137999 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zndl\" (UniqueName: \"kubernetes.io/projected/158becf0-d6cf-4917-8329-4f64ae5a4ca4-kube-api-access-4zndl\") pod \"thanos-querier-7c6f76d6d-k2h7h\" (UID: \"158becf0-d6cf-4917-8329-4f64ae5a4ca4\") " pod="openshift-monitoring/thanos-querier-7c6f76d6d-k2h7h" Oct 30 18:33:29.149895 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:29.149869 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7c6f76d6d-k2h7h" Oct 30 18:33:29.431731 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:29.431699 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7c6f76d6d-k2h7h"] Oct 30 18:33:29.433901 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:33:29.433870 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod158becf0_d6cf_4917_8329_4f64ae5a4ca4.slice/crio-119b97a32ab1fa4705dfa33313f5a74d8b7b63668d39e87f373a39bbe1dd070e WatchSource:0}: Error finding container 119b97a32ab1fa4705dfa33313f5a74d8b7b63668d39e87f373a39bbe1dd070e: Status 404 returned error can't find the container with id 119b97a32ab1fa4705dfa33313f5a74d8b7b63668d39e87f373a39bbe1dd070e Oct 30 18:33:29.439221 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:29.439197 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:29.542699 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:29.542674 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:29.579529 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:29.579500 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7gk7g" event={"ID":"4f8841c0-4cf3-45d9-9301-957d4fffe1ce","Type":"ContainerStarted","Data":"959bceca32a4648f175bb16a0a5c8bee7362c606e185799d119ebcc0d0ba185e"} Oct 30 18:33:29.579905 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:29.579566 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7gk7g" event={"ID":"4f8841c0-4cf3-45d9-9301-957d4fffe1ce","Type":"ContainerStarted","Data":"58759d94d79b7a8dfee02109cfbd5f7686de61c0c4a2c3fdd349eb6bc6973c34"} Oct 30 18:33:29.579905 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:29.579588 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7gk7g" event={"ID":"4f8841c0-4cf3-45d9-9301-957d4fffe1ce","Type":"ContainerStarted","Data":"af47a34bc4ee37645a7cb361ea43b89a72cf492fdc259bdc0a41add9a39429a8"} Oct 30 18:33:29.581690 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:29.581663 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hh862" event={"ID":"433ed181-b45e-45c0-a593-9cb0b6c09348","Type":"ContainerStarted","Data":"a99ba09dc8dff08ce6012ebe43f535bc1ccd84761030d6ded70f9edc886ab58e"} Oct 30 18:33:29.581795 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:29.581713 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hh862" event={"ID":"433ed181-b45e-45c0-a593-9cb0b6c09348","Type":"ContainerStarted","Data":"0e912d29e96fe3d0306e453d0573b73f28fffe45605db77ec4cbba8d1d320e7a"} Oct 30 18:33:29.583691 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:29.583668 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8698b" event={"ID":"312dae04-dfda-41da-90ca-36b3b44a12ea","Type":"ContainerStarted","Data":"a75be9204c2fe9a484cecb08bf4c302857165282b2b621cc491735af79e1e523"} Oct 30 18:33:29.584810 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:29.584790 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d","Type":"ContainerStarted","Data":"50b0bb5c3b75d6ed563d7acd7bc142c8e6e66f899b26bcdb63b7e097dea2f1fa"} Oct 30 18:33:29.585768 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:29.585750 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c6f76d6d-k2h7h" event={"ID":"158becf0-d6cf-4917-8329-4f64ae5a4ca4","Type":"ContainerStarted","Data":"119b97a32ab1fa4705dfa33313f5a74d8b7b63668d39e87f373a39bbe1dd070e"} Oct 30 18:33:29.608428 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:29.608384 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7gk7g" podStartSLOduration=2.184600508 podStartE2EDuration="3.608370454s" totalImagesPullingTime="1.423769946s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:33:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:33:29.608072951 +0000 UTC m=+47.827064376" watchObservedRunningTime="2026-10-30 18:33:29.608370454 +0000 UTC m=+47.827361877" Oct 30 18:33:29.635021 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:29.634978 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-hh862" podStartSLOduration=2.489273873 podStartE2EDuration="3.634968416s" totalImagesPullingTime="1.056555887s" totalInitContainerRuntime="89.138656ms" isStatefulPod=true podCreationTimestamp="2026-10-30 18:33:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:33:29.633698906 +0000 UTC m=+47.852690332" watchObservedRunningTime="2026-10-30 18:33:29.634968416 +0000 UTC m=+47.853959835" Oct 30 18:33:29.660771 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:29.660727 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8698b" podStartSLOduration=2.675750822 podStartE2EDuration="3.660713854s" totalImagesPullingTime="984.963032ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:33:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:33:29.659506543 +0000 UTC m=+47.878497965" watchObservedRunningTime="2026-10-30 18:33:29.660713854 +0000 UTC m=+47.879705282" Oct 30 18:33:29.836937 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:29.836849 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:29.940860 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:29.940820 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:30.047755 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:30.047727 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:30.151969 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:30.151937 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:30.592030 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:30.591992 2575 generic.go:356] "Generic (PLEG): container finished" podID="7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d" containerID="e01e0e8685fc3c23441371b7c14e6a97b36e570510e3628e07b8c3e5f4bf2e84" exitCode=0 Oct 30 18:33:30.592829 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:30.592696 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d","Type":"ContainerDied","Data":"e01e0e8685fc3c23441371b7c14e6a97b36e570510e3628e07b8c3e5f4bf2e84"} Oct 30 18:33:30.624313 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:30.624258 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:30.730696 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:30.730663 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:30.837363 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:30.837332 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:30.943182 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:30.943080 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:31.226536 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:31.226463 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-b6d7bcd58-z2xvj"] Oct 30 18:33:31.229644 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:31.229617 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-b6d7bcd58-z2xvj" Oct 30 18:33:31.238006 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:31.237730 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/2069a85a-44e7-4c98-9f47-7fad4416c15b-audit-log\") pod \"metrics-server-b6d7bcd58-z2xvj\" (UID: \"2069a85a-44e7-4c98-9f47-7fad4416c15b\") " pod="openshift-monitoring/metrics-server-b6d7bcd58-z2xvj" Oct 30 18:33:31.238006 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:31.237785 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wq2pp\" (UniqueName: \"kubernetes.io/projected/2069a85a-44e7-4c98-9f47-7fad4416c15b-kube-api-access-wq2pp\") pod \"metrics-server-b6d7bcd58-z2xvj\" (UID: \"2069a85a-44e7-4c98-9f47-7fad4416c15b\") " pod="openshift-monitoring/metrics-server-b6d7bcd58-z2xvj" Oct 30 18:33:31.238006 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:31.237820 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/2069a85a-44e7-4c98-9f47-7fad4416c15b-metrics-server-audit-profiles\") pod \"metrics-server-b6d7bcd58-z2xvj\" (UID: \"2069a85a-44e7-4c98-9f47-7fad4416c15b\") " pod="openshift-monitoring/metrics-server-b6d7bcd58-z2xvj" Oct 30 18:33:31.238006 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:31.237876 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2069a85a-44e7-4c98-9f47-7fad4416c15b-client-ca-bundle\") pod \"metrics-server-b6d7bcd58-z2xvj\" (UID: \"2069a85a-44e7-4c98-9f47-7fad4416c15b\") " pod="openshift-monitoring/metrics-server-b6d7bcd58-z2xvj" Oct 30 18:33:31.238006 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:31.237937 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/2069a85a-44e7-4c98-9f47-7fad4416c15b-secret-metrics-server-client-certs\") pod \"metrics-server-b6d7bcd58-z2xvj\" (UID: \"2069a85a-44e7-4c98-9f47-7fad4416c15b\") " pod="openshift-monitoring/metrics-server-b6d7bcd58-z2xvj" Oct 30 18:33:31.238006 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:31.237960 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/2069a85a-44e7-4c98-9f47-7fad4416c15b-secret-metrics-server-tls\") pod \"metrics-server-b6d7bcd58-z2xvj\" (UID: \"2069a85a-44e7-4c98-9f47-7fad4416c15b\") " pod="openshift-monitoring/metrics-server-b6d7bcd58-z2xvj" Oct 30 18:33:31.238006 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:31.237989 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2069a85a-44e7-4c98-9f47-7fad4416c15b-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-b6d7bcd58-z2xvj\" (UID: \"2069a85a-44e7-4c98-9f47-7fad4416c15b\") " pod="openshift-monitoring/metrics-server-b6d7bcd58-z2xvj" Oct 30 18:33:31.240631 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:31.240591 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 18:33:31.242014 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:31.241980 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 30 18:33:31.245858 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:31.245837 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-b6d7bcd58-z2xvj"] Oct 30 18:33:31.254918 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:31.254896 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 30 18:33:31.255186 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:31.255137 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 30 18:33:31.255271 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:31.255183 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-xsppx\"" Oct 30 18:33:31.256909 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:31.256889 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-958spucjgsqmv\"" Oct 30 18:33:31.339070 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:31.339040 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/2069a85a-44e7-4c98-9f47-7fad4416c15b-metrics-server-audit-profiles\") pod \"metrics-server-b6d7bcd58-z2xvj\" (UID: \"2069a85a-44e7-4c98-9f47-7fad4416c15b\") " pod="openshift-monitoring/metrics-server-b6d7bcd58-z2xvj" Oct 30 18:33:31.339257 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:31.339197 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2069a85a-44e7-4c98-9f47-7fad4416c15b-client-ca-bundle\") pod \"metrics-server-b6d7bcd58-z2xvj\" (UID: \"2069a85a-44e7-4c98-9f47-7fad4416c15b\") " pod="openshift-monitoring/metrics-server-b6d7bcd58-z2xvj" Oct 30 18:33:31.339325 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:31.339305 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/2069a85a-44e7-4c98-9f47-7fad4416c15b-secret-metrics-server-client-certs\") pod \"metrics-server-b6d7bcd58-z2xvj\" (UID: \"2069a85a-44e7-4c98-9f47-7fad4416c15b\") " pod="openshift-monitoring/metrics-server-b6d7bcd58-z2xvj" Oct 30 18:33:31.339406 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:31.339339 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/2069a85a-44e7-4c98-9f47-7fad4416c15b-secret-metrics-server-tls\") pod \"metrics-server-b6d7bcd58-z2xvj\" (UID: \"2069a85a-44e7-4c98-9f47-7fad4416c15b\") " pod="openshift-monitoring/metrics-server-b6d7bcd58-z2xvj" Oct 30 18:33:31.339518 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:31.339420 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2069a85a-44e7-4c98-9f47-7fad4416c15b-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-b6d7bcd58-z2xvj\" (UID: \"2069a85a-44e7-4c98-9f47-7fad4416c15b\") " pod="openshift-monitoring/metrics-server-b6d7bcd58-z2xvj" Oct 30 18:33:31.339518 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:31.339465 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/2069a85a-44e7-4c98-9f47-7fad4416c15b-audit-log\") pod \"metrics-server-b6d7bcd58-z2xvj\" (UID: \"2069a85a-44e7-4c98-9f47-7fad4416c15b\") " pod="openshift-monitoring/metrics-server-b6d7bcd58-z2xvj" Oct 30 18:33:31.339518 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:31.339513 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq2pp\" (UniqueName: \"kubernetes.io/projected/2069a85a-44e7-4c98-9f47-7fad4416c15b-kube-api-access-wq2pp\") pod \"metrics-server-b6d7bcd58-z2xvj\" (UID: \"2069a85a-44e7-4c98-9f47-7fad4416c15b\") " pod="openshift-monitoring/metrics-server-b6d7bcd58-z2xvj" Oct 30 18:33:31.340095 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:31.340074 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/2069a85a-44e7-4c98-9f47-7fad4416c15b-metrics-server-audit-profiles\") pod \"metrics-server-b6d7bcd58-z2xvj\" (UID: \"2069a85a-44e7-4c98-9f47-7fad4416c15b\") " pod="openshift-monitoring/metrics-server-b6d7bcd58-z2xvj" Oct 30 18:33:31.340842 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:31.340779 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/2069a85a-44e7-4c98-9f47-7fad4416c15b-audit-log\") pod \"metrics-server-b6d7bcd58-z2xvj\" (UID: \"2069a85a-44e7-4c98-9f47-7fad4416c15b\") " pod="openshift-monitoring/metrics-server-b6d7bcd58-z2xvj" Oct 30 18:33:31.341214 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:31.341188 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2069a85a-44e7-4c98-9f47-7fad4416c15b-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-b6d7bcd58-z2xvj\" (UID: \"2069a85a-44e7-4c98-9f47-7fad4416c15b\") " pod="openshift-monitoring/metrics-server-b6d7bcd58-z2xvj" Oct 30 18:33:31.342673 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:31.342650 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/2069a85a-44e7-4c98-9f47-7fad4416c15b-secret-metrics-server-tls\") pod \"metrics-server-b6d7bcd58-z2xvj\" (UID: \"2069a85a-44e7-4c98-9f47-7fad4416c15b\") " pod="openshift-monitoring/metrics-server-b6d7bcd58-z2xvj" Oct 30 18:33:31.343698 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:31.343676 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/2069a85a-44e7-4c98-9f47-7fad4416c15b-secret-metrics-server-client-certs\") pod \"metrics-server-b6d7bcd58-z2xvj\" (UID: \"2069a85a-44e7-4c98-9f47-7fad4416c15b\") " pod="openshift-monitoring/metrics-server-b6d7bcd58-z2xvj" Oct 30 18:33:31.344235 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:31.344204 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2069a85a-44e7-4c98-9f47-7fad4416c15b-client-ca-bundle\") pod \"metrics-server-b6d7bcd58-z2xvj\" (UID: \"2069a85a-44e7-4c98-9f47-7fad4416c15b\") " pod="openshift-monitoring/metrics-server-b6d7bcd58-z2xvj" Oct 30 18:33:31.353187 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:31.353118 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wq2pp\" (UniqueName: \"kubernetes.io/projected/2069a85a-44e7-4c98-9f47-7fad4416c15b-kube-api-access-wq2pp\") pod \"metrics-server-b6d7bcd58-z2xvj\" (UID: \"2069a85a-44e7-4c98-9f47-7fad4416c15b\") " pod="openshift-monitoring/metrics-server-b6d7bcd58-z2xvj" Oct 30 18:33:31.373934 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:31.373914 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:31.502075 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:31.502052 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:31.545327 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:31.545303 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-b6d7bcd58-z2xvj" Oct 30 18:33:31.595408 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:31.595374 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-r5d75"] Oct 30 18:33:31.603282 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:31.603255 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-r5d75" Oct 30 18:33:31.609585 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:31.609531 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-r5d75"] Oct 30 18:33:31.622431 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:31.622397 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c6f76d6d-k2h7h" event={"ID":"158becf0-d6cf-4917-8329-4f64ae5a4ca4","Type":"ContainerStarted","Data":"aa984b886f426081acc9f8e6bbb855480188d44ac8101f9c683c25a3d1652e63"} Oct 30 18:33:31.622560 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:31.622483 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c6f76d6d-k2h7h" event={"ID":"158becf0-d6cf-4917-8329-4f64ae5a4ca4","Type":"ContainerStarted","Data":"14ba34413e01d585a01b547969a283492df24ad5ce9f7ed233032765b9f9bf63"} Oct 30 18:33:31.622560 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:31.622526 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c6f76d6d-k2h7h" event={"ID":"158becf0-d6cf-4917-8329-4f64ae5a4ca4","Type":"ContainerStarted","Data":"5535e22ff5214af339308fb4fc3e670ed53f1f0f239c12b6005f9dc9bdc15be1"} Oct 30 18:33:31.626479 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:31.626409 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-4n8xm\"" Oct 30 18:33:31.626992 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:31.626921 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 30 18:33:31.630821 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:31.630767 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:31.646785 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:31.646657 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/25377c15-6e29-46b3-9f1b-9b1fc38570f5-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-r5d75\" (UID: \"25377c15-6e29-46b3-9f1b-9b1fc38570f5\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-r5d75" Oct 30 18:33:31.646785 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:31.646697 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6546m\" (UniqueName: \"kubernetes.io/projected/25377c15-6e29-46b3-9f1b-9b1fc38570f5-kube-api-access-6546m\") pod \"monitoring-plugin-77f4f76594-r5d75\" (UID: \"25377c15-6e29-46b3-9f1b-9b1fc38570f5\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-r5d75" Oct 30 18:33:31.747914 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:31.747834 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/25377c15-6e29-46b3-9f1b-9b1fc38570f5-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-r5d75\" (UID: \"25377c15-6e29-46b3-9f1b-9b1fc38570f5\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-r5d75" Oct 30 18:33:31.747914 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:31.747893 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6546m\" (UniqueName: \"kubernetes.io/projected/25377c15-6e29-46b3-9f1b-9b1fc38570f5-kube-api-access-6546m\") pod \"monitoring-plugin-77f4f76594-r5d75\" (UID: \"25377c15-6e29-46b3-9f1b-9b1fc38570f5\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-r5d75" Oct 30 18:33:31.750509 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:31.750475 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/25377c15-6e29-46b3-9f1b-9b1fc38570f5-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-r5d75\" (UID: \"25377c15-6e29-46b3-9f1b-9b1fc38570f5\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-r5d75" Oct 30 18:33:31.766069 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:31.766040 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6546m\" (UniqueName: \"kubernetes.io/projected/25377c15-6e29-46b3-9f1b-9b1fc38570f5-kube-api-access-6546m\") pod \"monitoring-plugin-77f4f76594-r5d75\" (UID: \"25377c15-6e29-46b3-9f1b-9b1fc38570f5\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-r5d75" Oct 30 18:33:31.798072 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:31.798026 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:31.832629 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:31.832490 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-b6d7bcd58-z2xvj"] Oct 30 18:33:31.835099 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:33:31.835073 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2069a85a_44e7_4c98_9f47_7fad4416c15b.slice/crio-f7b465a0d6ebcaf211c2d96f236da176e968470550f58af269fdd08aa306e298 WatchSource:0}: Error finding container f7b465a0d6ebcaf211c2d96f236da176e968470550f58af269fdd08aa306e298: Status 404 returned error can't find the container with id f7b465a0d6ebcaf211c2d96f236da176e968470550f58af269fdd08aa306e298 Oct 30 18:33:31.922631 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:31.922603 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-r5d75" Oct 30 18:33:32.205389 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:32.205192 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-r5d75"] Oct 30 18:33:32.207544 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:33:32.207518 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25377c15_6e29_46b3_9f1b_9b1fc38570f5.slice/crio-37d5c880b26121a7ab8d7ad2034e9521e55fd3ef0e30c419741bfabac5b7882a WatchSource:0}: Error finding container 37d5c880b26121a7ab8d7ad2034e9521e55fd3ef0e30c419741bfabac5b7882a: Status 404 returned error can't find the container with id 37d5c880b26121a7ab8d7ad2034e9521e55fd3ef0e30c419741bfabac5b7882a Oct 30 18:33:32.633370 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:32.633324 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c6f76d6d-k2h7h" event={"ID":"158becf0-d6cf-4917-8329-4f64ae5a4ca4","Type":"ContainerStarted","Data":"b293041f834a8c58a9afe73785f02d0d745e054eeeb364cabe34cfbeaecec38c"} Oct 30 18:33:32.633880 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:32.633414 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c6f76d6d-k2h7h" event={"ID":"158becf0-d6cf-4917-8329-4f64ae5a4ca4","Type":"ContainerStarted","Data":"a4ab3542fe23fa0a87f99732d545c84eefaceed9667bc8864e6fdbd0db0d05fc"} Oct 30 18:33:32.633880 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:32.633456 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c6f76d6d-k2h7h" event={"ID":"158becf0-d6cf-4917-8329-4f64ae5a4ca4","Type":"ContainerStarted","Data":"97d1910cdc430ab46adaee17577f1a58d8ecf4f5988e84eb0d52374a7784b62e"} Oct 30 18:33:32.634280 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:32.634248 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7c6f76d6d-k2h7h" Oct 30 18:33:32.635840 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:32.635815 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-r5d75" event={"ID":"25377c15-6e29-46b3-9f1b-9b1fc38570f5","Type":"ContainerStarted","Data":"37d5c880b26121a7ab8d7ad2034e9521e55fd3ef0e30c419741bfabac5b7882a"} Oct 30 18:33:32.640446 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:32.639592 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d","Type":"ContainerStarted","Data":"40e4c92d1f7f453ca5dd51f1e5ce8c9880c13a65c1a53e541e169a0f4f73d36e"} Oct 30 18:33:32.640446 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:32.639664 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d","Type":"ContainerStarted","Data":"f55ffe8e178a3ea54eee612711a21d86b51a2c7a8aea9865c7b94f0a63bfda88"} Oct 30 18:33:32.640446 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:32.639714 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d","Type":"ContainerStarted","Data":"45f502123b296292940de2da4674a9bd8550f926d07d158169e92083eec6a1d1"} Oct 30 18:33:32.640446 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:32.639753 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d","Type":"ContainerStarted","Data":"150cb91b1240c1b880b23a5afdccd8da7736b40f5173f9d83e18b5551baf7d5c"} Oct 30 18:33:32.640446 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:32.639795 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d","Type":"ContainerStarted","Data":"28b9e7d1d345ce49931ddbd2d6c992e4a2be94d3a1c3410c8ecea6cfaf4329da"} Oct 30 18:33:32.640446 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:32.639832 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d","Type":"ContainerStarted","Data":"3fc0ad5914603e602d4c8f2e36d32315d805175b0b3c3bc842e807ce226d5ff7"} Oct 30 18:33:32.640821 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:32.640800 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-b6d7bcd58-z2xvj" event={"ID":"2069a85a-44e7-4c98-9f47-7fad4416c15b","Type":"ContainerStarted","Data":"f7b465a0d6ebcaf211c2d96f236da176e968470550f58af269fdd08aa306e298"} Oct 30 18:33:32.685244 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:32.685173 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7c6f76d6d-k2h7h" podStartSLOduration=1.970771708 podStartE2EDuration="4.685126371s" totalImagesPullingTime="2.714354663s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:33:28 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:33:32.683587168 +0000 UTC m=+50.902578591" watchObservedRunningTime="2026-10-30 18:33:32.685126371 +0000 UTC m=+50.904117797" Oct 30 18:33:32.718568 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:32.718513 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.263030929 podStartE2EDuration="5.718498936s" totalImagesPullingTime="2.442291205s" totalInitContainerRuntime="13.176802ms" isStatefulPod=false podCreationTimestamp="2026-10-30 18:33:27 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:33:32.716644236 +0000 UTC m=+50.935635662" watchObservedRunningTime="2026-10-30 18:33:32.718498936 +0000 UTC m=+50.937490352" Oct 30 18:33:33.281642 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:33.281609 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 18:33:33.286000 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:33.285971 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:33.299649 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:33.299607 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 18:33:33.299649 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:33.299630 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 18:33:33.299649 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:33.299637 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 18:33:33.305890 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:33.305863 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 18:33:33.306031 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:33.305933 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 18:33:33.306088 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:33.306040 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 18:33:33.306088 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:33.306058 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 18:33:33.306088 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:33.306064 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 18:33:33.306211 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:33.306118 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-8b4gujb9r0sdv\"" Oct 30 18:33:33.307984 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:33.307916 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 18:33:33.308100 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:33.308013 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 18:33:33.308100 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:33.308015 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 18:33:33.308229 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:33.308107 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 18:33:33.308695 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:33.308671 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-5vglk\"" Oct 30 18:33:33.310195 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:33.309940 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 18:33:33.310195 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:33.309982 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 18:33:33.310322 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:33.310188 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:33.312018 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:33.311997 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 18:33:33.362637 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:33.362608 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8f76499e-97ad-4c03-8a72-a62f65619596-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:33.362797 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:33.362645 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8f76499e-97ad-4c03-8a72-a62f65619596-config\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:33.362797 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:33.362663 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8f76499e-97ad-4c03-8a72-a62f65619596-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:33.362797 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:33.362683 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f76499e-97ad-4c03-8a72-a62f65619596-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:33.362967 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:33.362812 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8f76499e-97ad-4c03-8a72-a62f65619596-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:33.362967 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:33.362839 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8f76499e-97ad-4c03-8a72-a62f65619596-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:33.362967 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:33.362874 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrzxm\" (UniqueName: \"kubernetes.io/projected/8f76499e-97ad-4c03-8a72-a62f65619596-kube-api-access-jrzxm\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:33.362967 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:33.362895 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8f76499e-97ad-4c03-8a72-a62f65619596-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:33.362967 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:33.362949 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8f76499e-97ad-4c03-8a72-a62f65619596-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:33.363239 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:33.362998 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f76499e-97ad-4c03-8a72-a62f65619596-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:33.363239 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:33.363032 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8f76499e-97ad-4c03-8a72-a62f65619596-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:33.363239 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:33.363157 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8f76499e-97ad-4c03-8a72-a62f65619596-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:33.363239 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:33.363194 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8f76499e-97ad-4c03-8a72-a62f65619596-web-config\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:33.363239 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:33.363221 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8f76499e-97ad-4c03-8a72-a62f65619596-config-out\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:33.363483 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:33.363242 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8f76499e-97ad-4c03-8a72-a62f65619596-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:33.363483 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:33.363286 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8f76499e-97ad-4c03-8a72-a62f65619596-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:33.363483 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:33.363334 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8f76499e-97ad-4c03-8a72-a62f65619596-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:33.363483 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:33.363387 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f76499e-97ad-4c03-8a72-a62f65619596-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:33.363483 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:33.363413 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8f76499e-97ad-4c03-8a72-a62f65619596-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:33.363483 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:33.363431 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8f76499e-97ad-4c03-8a72-a62f65619596-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:33.464813 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:33.464788 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8f76499e-97ad-4c03-8a72-a62f65619596-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:33.464933 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:33.464819 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8f76499e-97ad-4c03-8a72-a62f65619596-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:33.464933 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:33.464860 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f76499e-97ad-4c03-8a72-a62f65619596-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:33.464933 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:33.464888 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8f76499e-97ad-4c03-8a72-a62f65619596-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:33.464933 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:33.464913 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8f76499e-97ad-4c03-8a72-a62f65619596-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:33.465196 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:33.464956 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8f76499e-97ad-4c03-8a72-a62f65619596-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:33.465196 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:33.464988 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8f76499e-97ad-4c03-8a72-a62f65619596-config\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:33.465196 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:33.465012 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8f76499e-97ad-4c03-8a72-a62f65619596-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:33.465196 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:33.465040 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f76499e-97ad-4c03-8a72-a62f65619596-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:33.465196 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:33.465075 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8f76499e-97ad-4c03-8a72-a62f65619596-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:33.465196 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:33.465097 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8f76499e-97ad-4c03-8a72-a62f65619596-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:33.465196 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:33.465122 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrzxm\" (UniqueName: \"kubernetes.io/projected/8f76499e-97ad-4c03-8a72-a62f65619596-kube-api-access-jrzxm\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:33.465196 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:33.465168 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8f76499e-97ad-4c03-8a72-a62f65619596-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:33.465196 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:33.465193 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8f76499e-97ad-4c03-8a72-a62f65619596-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:33.465831 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:33.465226 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f76499e-97ad-4c03-8a72-a62f65619596-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:33.465831 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:33.465250 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8f76499e-97ad-4c03-8a72-a62f65619596-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:33.465831 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:33.465291 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8f76499e-97ad-4c03-8a72-a62f65619596-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:33.465831 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:33.465333 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8f76499e-97ad-4c03-8a72-a62f65619596-web-config\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:33.465831 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:33.465384 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8f76499e-97ad-4c03-8a72-a62f65619596-config-out\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:33.465831 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:33.465408 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8f76499e-97ad-4c03-8a72-a62f65619596-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:33.466128 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:33.465883 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f76499e-97ad-4c03-8a72-a62f65619596-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:33.466443 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:33.466419 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8f76499e-97ad-4c03-8a72-a62f65619596-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:33.467279 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:33.467250 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8f76499e-97ad-4c03-8a72-a62f65619596-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:33.467799 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:33.467774 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8f76499e-97ad-4c03-8a72-a62f65619596-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:33.468216 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:33.468191 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f76499e-97ad-4c03-8a72-a62f65619596-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:33.468488 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:33.468458 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8f76499e-97ad-4c03-8a72-a62f65619596-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:33.468852 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:33.468831 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8f76499e-97ad-4c03-8a72-a62f65619596-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:33.469319 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:33.469187 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8f76499e-97ad-4c03-8a72-a62f65619596-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:33.469516 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:33.469491 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f76499e-97ad-4c03-8a72-a62f65619596-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:33.469586 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:33.469557 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8f76499e-97ad-4c03-8a72-a62f65619596-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:33.470025 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:33.470004 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8f76499e-97ad-4c03-8a72-a62f65619596-web-config\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:33.471496 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:33.471469 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8f76499e-97ad-4c03-8a72-a62f65619596-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:33.472781 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:33.472761 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8f76499e-97ad-4c03-8a72-a62f65619596-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:33.472984 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:33.472948 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8f76499e-97ad-4c03-8a72-a62f65619596-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:33.473091 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:33.473075 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8f76499e-97ad-4c03-8a72-a62f65619596-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:33.473582 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:33.473562 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8f76499e-97ad-4c03-8a72-a62f65619596-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:33.473667 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:33.473607 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8f76499e-97ad-4c03-8a72-a62f65619596-config-out\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:33.474010 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:33.473989 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8f76499e-97ad-4c03-8a72-a62f65619596-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:33.475059 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:33.475037 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/8f76499e-97ad-4c03-8a72-a62f65619596-config\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:33.480183 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:33.480139 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrzxm\" (UniqueName: \"kubernetes.io/projected/8f76499e-97ad-4c03-8a72-a62f65619596-kube-api-access-jrzxm\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:33.617912 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:33.617883 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:33.645666 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:33.645631 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-b6d7bcd58-z2xvj" event={"ID":"2069a85a-44e7-4c98-9f47-7fad4416c15b","Type":"ContainerStarted","Data":"54c9cd22267da3d77e5bca53808fc9fa6c90cd233d8eb7e6b8fb1f96c59a1e23"} Oct 30 18:33:33.647156 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:33.647114 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-r5d75" event={"ID":"25377c15-6e29-46b3-9f1b-9b1fc38570f5","Type":"ContainerStarted","Data":"be487c3ee3ee677053877d0f2e12755dca0dc772ce85c179941d9432b95c34c7"} Oct 30 18:33:33.647849 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:33.647809 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-r5d75" Oct 30 18:33:33.652070 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:33.652027 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-r5d75" Oct 30 18:33:33.695283 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:33.695123 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-r5d75" podStartSLOduration=1.350442015 podStartE2EDuration="2.695108767s" totalImagesPullingTime="1.344666752s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:33:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:33:33.694226785 +0000 UTC m=+51.913218208" watchObservedRunningTime="2026-10-30 18:33:33.695108767 +0000 UTC m=+51.914100188" Oct 30 18:33:33.695670 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:33.695649 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-b6d7bcd58-z2xvj" podStartSLOduration=1.136734459 podStartE2EDuration="2.695641366s" totalImagesPullingTime="1.558906907s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:33:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:33:33.674380539 +0000 UTC m=+51.893371962" watchObservedRunningTime="2026-10-30 18:33:33.695641366 +0000 UTC m=+51.914632791" Oct 30 18:33:33.917858 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:33.917724 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 18:33:33.919848 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:33:33.919823 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f76499e_97ad_4c03_8a72_a62f65619596.slice/crio-32522431fb329153a6eeebccc7680bd24dcdea651248be4cc3d5f50b8bb5a1e3 WatchSource:0}: Error finding container 32522431fb329153a6eeebccc7680bd24dcdea651248be4cc3d5f50b8bb5a1e3: Status 404 returned error can't find the container with id 32522431fb329153a6eeebccc7680bd24dcdea651248be4cc3d5f50b8bb5a1e3 Oct 30 18:33:34.652640 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:34.652604 2575 generic.go:356] "Generic (PLEG): container finished" podID="8f76499e-97ad-4c03-8a72-a62f65619596" containerID="696e8840b46d4d1acf76b61bafc7bc0d953821760feec22065f848f3daa8eaf7" exitCode=0 Oct 30 18:33:34.653106 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:34.652704 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8f76499e-97ad-4c03-8a72-a62f65619596","Type":"ContainerDied","Data":"696e8840b46d4d1acf76b61bafc7bc0d953821760feec22065f848f3daa8eaf7"} Oct 30 18:33:34.653106 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:34.652791 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8f76499e-97ad-4c03-8a72-a62f65619596","Type":"ContainerStarted","Data":"32522431fb329153a6eeebccc7680bd24dcdea651248be4cc3d5f50b8bb5a1e3"} Oct 30 18:33:34.693484 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:34.693429 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:34.813041 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:34.813014 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:36.480740 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:36.480711 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:36.571926 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:36.571896 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-vfn4g" Oct 30 18:33:36.590506 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:36.590478 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:37.381135 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:37.381062 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:37.496190 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:37.496159 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:37.668177 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:37.668124 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8f76499e-97ad-4c03-8a72-a62f65619596","Type":"ContainerStarted","Data":"c5aea0259e406d5b190a03fce661f9a4052cc4d10160e32c100ffdf12f7433d5"} Oct 30 18:33:38.674945 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:38.674903 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8f76499e-97ad-4c03-8a72-a62f65619596","Type":"ContainerStarted","Data":"563244759dbdbc2c58d0b4ba9e94b736d2bb8e69feb6e845291e2e2d8bf3fcb2"} Oct 30 18:33:38.675320 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:38.674966 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8f76499e-97ad-4c03-8a72-a62f65619596","Type":"ContainerStarted","Data":"be397bdaf885f8c055c80395dea268c2f30e89f2cb1526720d2440af75217759"} Oct 30 18:33:38.675320 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:38.674992 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8f76499e-97ad-4c03-8a72-a62f65619596","Type":"ContainerStarted","Data":"2b7e0cbea7cc9adfbc4ddedcd2bb9ab689149d004cf05ae4c38d9a2c2ec563f5"} Oct 30 18:33:38.675320 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:38.675016 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8f76499e-97ad-4c03-8a72-a62f65619596","Type":"ContainerStarted","Data":"a7ec88732cafa80ab354e8864f75c1833177871423f5ca944fbf24dc7bb44a2b"} Oct 30 18:33:38.675320 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:38.675040 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8f76499e-97ad-4c03-8a72-a62f65619596","Type":"ContainerStarted","Data":"b049f7e3597139cac61bd3d2c74278bbfd9a2df99c6ffeaef14d0bda8e236eef"} Oct 30 18:33:38.716980 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:38.716924 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.791425469 podStartE2EDuration="5.71690631s" totalImagesPullingTime="2.910851112s" totalInitContainerRuntime="14.629729ms" isStatefulPod=false podCreationTimestamp="2026-10-30 18:33:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:33:38.71641901 +0000 UTC m=+56.935410435" watchObservedRunningTime="2026-10-30 18:33:38.71690631 +0000 UTC m=+56.935897732" Oct 30 18:33:39.605648 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:39.605616 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:39.658827 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:39.658798 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7c6f76d6d-k2h7h" Oct 30 18:33:39.723082 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:39.723049 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:40.258496 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:40.258470 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:40.459890 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:40.459862 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:42.524814 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:42.524790 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:33:42.711962 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:42.711915 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:42.816124 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:42.816091 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:43.619007 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:43.618960 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:46.316373 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:46.316336 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:46.439811 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:46.439782 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:47.082739 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:47.082708 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e163025a-b19f-44e2-9058-b65f31bfbcc4-original-pull-secret\") pod \"global-pull-secret-syncer-zmdjg\" (UID: \"e163025a-b19f-44e2-9058-b65f31bfbcc4\") " pod="kube-system/global-pull-secret-syncer-zmdjg" Oct 30 18:33:47.082909 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:47.082752 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90488e60-cb70-4afa-a54b-6f333d345e84-metrics-certs\") pod \"network-metrics-daemon-dtppz\" (UID: \"90488e60-cb70-4afa-a54b-6f333d345e84\") " pod="openshift-multus/network-metrics-daemon-dtppz" Oct 30 18:33:47.086487 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:47.086464 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 18:33:47.086590 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:47.086520 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 18:33:47.095903 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:47.095881 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e163025a-b19f-44e2-9058-b65f31bfbcc4-original-pull-secret\") pod \"global-pull-secret-syncer-zmdjg\" (UID: \"e163025a-b19f-44e2-9058-b65f31bfbcc4\") " pod="kube-system/global-pull-secret-syncer-zmdjg" Oct 30 18:33:47.096020 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:47.095997 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90488e60-cb70-4afa-a54b-6f333d345e84-metrics-certs\") pod \"network-metrics-daemon-dtppz\" (UID: \"90488e60-cb70-4afa-a54b-6f333d345e84\") " pod="openshift-multus/network-metrics-daemon-dtppz" Oct 30 18:33:47.123542 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:47.122460 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-z87v4\"" Oct 30 18:33:47.127354 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:47.127331 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dtppz" Oct 30 18:33:47.183859 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:47.183830 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztd9j\" (UniqueName: \"kubernetes.io/projected/fb66d280-cf24-4549-8fdb-89f16b6080c1-kube-api-access-ztd9j\") pod \"network-check-target-26f57\" (UID: \"fb66d280-cf24-4549-8fdb-89f16b6080c1\") " pod="openshift-network-diagnostics/network-check-target-26f57" Oct 30 18:33:47.187492 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:47.187470 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 18:33:47.197637 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:47.197576 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 18:33:47.218593 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:47.218214 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztd9j\" (UniqueName: \"kubernetes.io/projected/fb66d280-cf24-4549-8fdb-89f16b6080c1-kube-api-access-ztd9j\") pod \"network-check-target-26f57\" (UID: \"fb66d280-cf24-4549-8fdb-89f16b6080c1\") " pod="openshift-network-diagnostics/network-check-target-26f57" Oct 30 18:33:47.220162 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:47.220126 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-9qttr\"" Oct 30 18:33:47.227354 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:47.226923 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zmdjg" Oct 30 18:33:47.227354 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:47.227184 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-fgz4w\"" Oct 30 18:33:47.234495 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:47.234470 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26f57" Oct 30 18:33:47.424204 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:47.424173 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dtppz"] Oct 30 18:33:47.427206 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:33:47.427159 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90488e60_cb70_4afa_a54b_6f333d345e84.slice/crio-b40da5c1f34f4f94eb250f3bed5492801d825f9ce4f5fe472b32b5fae787bd59 WatchSource:0}: Error finding container b40da5c1f34f4f94eb250f3bed5492801d825f9ce4f5fe472b32b5fae787bd59: Status 404 returned error can't find the container with id b40da5c1f34f4f94eb250f3bed5492801d825f9ce4f5fe472b32b5fae787bd59 Oct 30 18:33:47.544845 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:47.544812 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zmdjg"] Oct 30 18:33:47.569422 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:47.565666 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-26f57"] Oct 30 18:33:47.581183 ip-10-0-140-163 kubenswrapper[2575]: W1030 18:33:47.581134 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb66d280_cf24_4549_8fdb_89f16b6080c1.slice/crio-06687992306a30ad3c6f9044932282015ccabc988888a1c185e89fa381425904 WatchSource:0}: Error finding container 06687992306a30ad3c6f9044932282015ccabc988888a1c185e89fa381425904: Status 404 returned error can't find the container with id 06687992306a30ad3c6f9044932282015ccabc988888a1c185e89fa381425904 Oct 30 18:33:47.719067 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:47.719034 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dtppz" event={"ID":"90488e60-cb70-4afa-a54b-6f333d345e84","Type":"ContainerStarted","Data":"b40da5c1f34f4f94eb250f3bed5492801d825f9ce4f5fe472b32b5fae787bd59"} Oct 30 18:33:47.720374 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:47.720350 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-26f57" event={"ID":"fb66d280-cf24-4549-8fdb-89f16b6080c1","Type":"ContainerStarted","Data":"06687992306a30ad3c6f9044932282015ccabc988888a1c185e89fa381425904"} Oct 30 18:33:47.721595 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:47.721554 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zmdjg" event={"ID":"e163025a-b19f-44e2-9058-b65f31bfbcc4","Type":"ContainerStarted","Data":"f4c144adff68eb30cf1418f5b0c943b73d737798daeb89a49a0e73db0db3b1db"} Oct 30 18:33:48.735915 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:48.735878 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dtppz" event={"ID":"90488e60-cb70-4afa-a54b-6f333d345e84","Type":"ContainerStarted","Data":"7ce88faa74ea52306181e4e47645e7bf386fa8870bd2157a938e79677fd0465b"} Oct 30 18:33:49.425663 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:49.425623 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:49.565646 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:49.565610 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:49.744219 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:49.743651 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dtppz" event={"ID":"90488e60-cb70-4afa-a54b-6f333d345e84","Type":"ContainerStarted","Data":"a5a8ac22f785439e02ddb6a679832008204754127308eb323c618b50f0e8ee22"} Oct 30 18:33:49.781740 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:49.781691 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-dtppz" podStartSLOduration=65.635053399 podStartE2EDuration="1m6.781677088s" totalImagesPullingTime="1.146623689s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:32:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:33:49.778249692 +0000 UTC m=+67.997241115" watchObservedRunningTime="2026-10-30 18:33:49.781677088 +0000 UTC m=+68.000668510" Oct 30 18:33:49.947371 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:49.947339 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:50.069738 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:50.069692 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:51.546293 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:51.546247 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-b6d7bcd58-z2xvj" Oct 30 18:33:51.546743 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:51.546323 2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-b6d7bcd58-z2xvj" Oct 30 18:33:51.753853 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:51.753816 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-26f57" event={"ID":"fb66d280-cf24-4549-8fdb-89f16b6080c1","Type":"ContainerStarted","Data":"0fa412b13c56b1d720d61cbdd900e0de2fc9efa1b155fe274a259aa643077c34"} Oct 30 18:33:51.754007 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:51.753913 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-26f57" Oct 30 18:33:51.775428 ip-10-0-140-163 kubenswrapper[2575]: I1030 18:33:51.775379 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-26f57" podStartSLOduration=65.563104867 podStartE2EDuration="1m8.775365652s" totalImagesPullingTime="3.212260785s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:32:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:33:51.775016534 +0000 UTC m=+69.994007961" watchObservedRunningTime="2026-10-30 18:33:51.775365652 +0000 UTC m=+69.994357074" Oct 30 18:33:52.408191 ip-10-0-140-163 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 18:33:52.430475 ip-10-0-140-163 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 18:33:52.430640 ip-10-0-140-163 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 18:33:52.430817 ip-10-0-140-163 systemd[1]: kubelet.service: Consumed 4.528s CPU time. Oct 30 18:33:52.438834 ip-10-0-140-163 systemd[1]: Starting Kubernetes Kubelet... Oct 30 18:33:52.502507 ip-10-0-140-163 kubenswrapper[6960]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 18:33:52.502507 ip-10-0-140-163 kubenswrapper[6960]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 18:33:52.502507 ip-10-0-140-163 kubenswrapper[6960]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 18:33:52.502507 ip-10-0-140-163 kubenswrapper[6960]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 18:33:52.504511 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504440 6960 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:33:52.504511 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504510 6960 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:33:52.504511 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504514 6960 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:33:52.504609 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504518 6960 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:33:52.504609 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504522 6960 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:33:52.504609 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504525 6960 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:33:52.504609 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504529 6960 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:33:52.504609 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504532 6960 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:33:52.504609 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504535 6960 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:33:52.504609 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504538 6960 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:33:52.504609 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504541 6960 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:33:52.504609 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504544 6960 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:33:52.504609 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504547 6960 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:33:52.504609 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504550 6960 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:33:52.504609 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504560 6960 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:33:52.504609 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504563 6960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:33:52.504609 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504566 6960 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:33:52.504609 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504569 6960 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:33:52.504609 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504571 6960 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:33:52.504609 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504574 6960 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:33:52.504609 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504577 6960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:33:52.504609 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504580 6960 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:33:52.505054 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504582 6960 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:33:52.505054 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504585 6960 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:33:52.505054 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504588 6960 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:33:52.505054 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504590 6960 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:33:52.505054 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504593 6960 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:33:52.505054 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504596 6960 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:33:52.505054 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504598 6960 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:33:52.505054 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504601 6960 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:33:52.505054 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504604 6960 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:33:52.505054 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504606 6960 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:33:52.505054 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504612 6960 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:33:52.505054 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504615 6960 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:33:52.505054 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504618 6960 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:33:52.505054 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504620 6960 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:33:52.505054 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504623 6960 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:33:52.505054 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504626 6960 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:33:52.505054 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504628 6960 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:33:52.505054 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504631 6960 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:33:52.505054 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504634 6960 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:33:52.505054 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504636 6960 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:33:52.505054 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504639 6960 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:33:52.505568 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504642 6960 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:33:52.505568 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504644 6960 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:33:52.505568 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504647 6960 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:33:52.505568 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504649 6960 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:33:52.505568 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504652 6960 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:33:52.505568 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504655 6960 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:33:52.505568 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504657 6960 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:33:52.505568 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504660 6960 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:33:52.505568 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504662 6960 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:33:52.505568 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504666 6960 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:33:52.505568 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504668 6960 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:33:52.505568 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504671 6960 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:33:52.505568 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504674 6960 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:33:52.505568 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504677 6960 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:33:52.505568 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504679 6960 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:33:52.505568 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504682 6960 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:33:52.505568 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504684 6960 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:33:52.505568 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504687 6960 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:33:52.505568 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504690 6960 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:33:52.505568 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504692 6960 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:33:52.506051 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504694 6960 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:33:52.506051 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504697 6960 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:33:52.506051 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504700 6960 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:33:52.506051 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504703 6960 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:33:52.506051 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504705 6960 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:33:52.506051 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504708 6960 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:33:52.506051 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504710 6960 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:33:52.506051 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504714 6960 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:33:52.506051 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504719 6960 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:33:52.506051 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504722 6960 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:33:52.506051 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504724 6960 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:33:52.506051 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504727 6960 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:33:52.506051 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504730 6960 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:33:52.506051 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504732 6960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:33:52.506051 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504736 6960 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:33:52.506051 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504739 6960 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:33:52.506051 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504742 6960 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:33:52.506051 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504744 6960 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:33:52.506051 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504747 6960 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:33:52.506548 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504749 6960 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:33:52.506548 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504752 6960 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:33:52.506548 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504755 6960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:33:52.506548 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504758 6960 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:33:52.506548 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504761 6960 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:33:52.506548 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504764 6960 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:33:52.506548 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504767 6960 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:33:52.506548 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504769 6960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:33:52.506548 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504773 6960 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:33:52.506548 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504777 6960 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:33:52.506548 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504780 6960 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:33:52.506548 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504784 6960 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:33:52.506548 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504787 6960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:33:52.506548 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504790 6960 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:33:52.506548 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504792 6960 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:33:52.506548 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504795 6960 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:33:52.506548 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504798 6960 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:33:52.506548 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504801 6960 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:33:52.506548 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504804 6960 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:33:52.507003 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504806 6960 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:33:52.507003 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504809 6960 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:33:52.507003 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504812 6960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:33:52.507003 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504815 6960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:33:52.507003 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.504818 6960 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:33:52.507003 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505606 6960 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:33:52.507003 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505613 6960 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:33:52.507003 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505616 6960 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:33:52.507003 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505619 6960 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:33:52.507003 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505622 6960 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:33:52.507003 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505624 6960 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:33:52.507003 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505627 6960 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:33:52.507003 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505630 6960 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:33:52.507003 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505633 6960 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:33:52.507003 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505636 6960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:33:52.507003 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505638 6960 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:33:52.507003 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505641 6960 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:33:52.507003 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505644 6960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:33:52.507003 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505647 6960 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:33:52.507496 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505649 6960 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:33:52.507496 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505652 6960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:33:52.507496 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505654 6960 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:33:52.507496 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505657 6960 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:33:52.507496 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505660 6960 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:33:52.507496 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505662 6960 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:33:52.507496 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505666 6960 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:33:52.507496 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505669 6960 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:33:52.507496 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505671 6960 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:33:52.507496 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505674 6960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:33:52.507496 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505677 6960 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:33:52.507496 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505682 6960 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:33:52.507496 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505685 6960 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:33:52.507496 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505688 6960 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:33:52.507496 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505691 6960 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:33:52.507496 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505694 6960 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:33:52.507496 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505698 6960 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:33:52.507496 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505702 6960 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:33:52.507496 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505705 6960 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:33:52.507971 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505708 6960 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:33:52.507971 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505710 6960 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:33:52.507971 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505714 6960 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:33:52.507971 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505718 6960 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:33:52.507971 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505721 6960 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:33:52.507971 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505724 6960 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:33:52.507971 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505727 6960 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:33:52.507971 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505730 6960 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:33:52.507971 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505732 6960 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:33:52.507971 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505735 6960 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:33:52.507971 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505738 6960 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:33:52.507971 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505740 6960 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:33:52.507971 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505744 6960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:33:52.507971 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505746 6960 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:33:52.507971 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505749 6960 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:33:52.507971 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505752 6960 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:33:52.507971 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505754 6960 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:33:52.507971 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505757 6960 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:33:52.508416 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505760 6960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:33:52.508416 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505763 6960 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:33:52.508416 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505766 6960 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:33:52.508416 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505768 6960 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:33:52.508416 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505771 6960 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:33:52.508416 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505774 6960 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:33:52.508416 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505777 6960 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:33:52.508416 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505780 6960 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:33:52.508416 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505783 6960 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:33:52.508416 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505785 6960 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:33:52.508416 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505788 6960 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:33:52.508416 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505790 6960 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:33:52.508416 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505793 6960 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:33:52.508416 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505795 6960 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:33:52.508416 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505798 6960 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:33:52.508416 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505801 6960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:33:52.508416 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505804 6960 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:33:52.508416 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505807 6960 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:33:52.508416 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505809 6960 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:33:52.508416 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505812 6960 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:33:52.508888 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505815 6960 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:33:52.508888 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505818 6960 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:33:52.508888 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505820 6960 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:33:52.508888 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505825 6960 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:33:52.508888 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505828 6960 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:33:52.508888 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505831 6960 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:33:52.508888 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505834 6960 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:33:52.508888 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505836 6960 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:33:52.508888 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505839 6960 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:33:52.508888 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505841 6960 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:33:52.508888 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505844 6960 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:33:52.508888 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505846 6960 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:33:52.508888 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505849 6960 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:33:52.508888 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505852 6960 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:33:52.508888 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505854 6960 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:33:52.508888 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505857 6960 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:33:52.508888 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505859 6960 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:33:52.508888 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505862 6960 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:33:52.508888 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505865 6960 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:33:52.508888 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505867 6960 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:33:52.509390 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505870 6960 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:33:52.509390 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505873 6960 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:33:52.509390 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505876 6960 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:33:52.509390 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505878 6960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:33:52.509390 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505881 6960 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:33:52.509390 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505883 6960 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:33:52.509390 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505886 6960 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:33:52.509390 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505889 6960 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:33:52.509390 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505891 6960 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:33:52.509390 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505894 6960 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:33:52.509390 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505896 6960 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:33:52.509390 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505900 6960 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:33:52.509390 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505903 6960 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:33:52.509390 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505906 6960 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:33:52.509390 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.505908 6960 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:33:52.509390 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506390 6960 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:33:52.509390 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506396 6960 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:33:52.509390 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506400 6960 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:33:52.509390 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506402 6960 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:33:52.509390 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506405 6960 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:33:52.509905 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506408 6960 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:33:52.509905 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506410 6960 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:33:52.509905 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506413 6960 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:33:52.509905 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506416 6960 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:33:52.509905 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506419 6960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:33:52.509905 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506422 6960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:33:52.509905 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506424 6960 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:33:52.509905 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506428 6960 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:33:52.509905 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506432 6960 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:33:52.509905 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506435 6960 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:33:52.509905 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506437 6960 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:33:52.509905 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506440 6960 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:33:52.509905 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506444 6960 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:33:52.509905 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506446 6960 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:33:52.509905 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506449 6960 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:33:52.509905 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506451 6960 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:33:52.509905 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506454 6960 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:33:52.509905 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506457 6960 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:33:52.509905 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506460 6960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:33:52.510397 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506462 6960 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:33:52.510397 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506465 6960 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:33:52.510397 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506468 6960 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:33:52.510397 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506470 6960 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:33:52.510397 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506473 6960 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:33:52.510397 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506476 6960 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:33:52.510397 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506479 6960 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:33:52.510397 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506481 6960 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:33:52.510397 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506486 6960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:33:52.510397 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506489 6960 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:33:52.510397 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506491 6960 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:33:52.510397 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506494 6960 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:33:52.510397 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506496 6960 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:33:52.510397 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506499 6960 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:33:52.510397 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506501 6960 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:33:52.510397 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506504 6960 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:33:52.510397 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506506 6960 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:33:52.510397 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506509 6960 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:33:52.510397 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506511 6960 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:33:52.510397 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506514 6960 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:33:52.511032 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506517 6960 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:33:52.511032 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506519 6960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:33:52.511032 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506522 6960 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:33:52.511032 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506524 6960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:33:52.511032 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506527 6960 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:33:52.511032 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506530 6960 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:33:52.511032 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506532 6960 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:33:52.511032 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506535 6960 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:33:52.511032 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506538 6960 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:33:52.511032 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506540 6960 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:33:52.511032 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506543 6960 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:33:52.511032 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506545 6960 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:33:52.511032 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506548 6960 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:33:52.511032 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506553 6960 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:33:52.511032 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506556 6960 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:33:52.511032 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506559 6960 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:33:52.511032 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506561 6960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:33:52.511032 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506564 6960 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:33:52.511032 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506566 6960 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:33:52.511032 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506569 6960 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:33:52.511893 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506573 6960 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:33:52.511893 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506575 6960 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:33:52.511893 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506578 6960 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:33:52.511893 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506583 6960 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:33:52.511893 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506587 6960 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:33:52.511893 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506590 6960 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:33:52.511893 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506594 6960 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:33:52.511893 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506596 6960 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:33:52.511893 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506599 6960 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:33:52.511893 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506601 6960 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:33:52.511893 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506604 6960 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:33:52.511893 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506606 6960 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:33:52.511893 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506609 6960 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:33:52.511893 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506612 6960 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:33:52.511893 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506615 6960 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:33:52.511893 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506617 6960 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:33:52.511893 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506620 6960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:33:52.511893 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506623 6960 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:33:52.511893 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506625 6960 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:33:52.512385 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506628 6960 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:33:52.512385 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506631 6960 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:33:52.512385 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506633 6960 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:33:52.512385 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506636 6960 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:33:52.512385 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506639 6960 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:33:52.512385 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506641 6960 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:33:52.512385 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506645 6960 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:33:52.512385 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506647 6960 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:33:52.512385 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506650 6960 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:33:52.512385 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506653 6960 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:33:52.512385 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506655 6960 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:33:52.512385 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506658 6960 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:33:52.512385 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506661 6960 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:33:52.512385 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506663 6960 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:33:52.512385 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506666 6960 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:33:52.512385 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506669 6960 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:33:52.512385 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506671 6960 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:33:52.512385 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506674 6960 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:33:52.512385 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506677 6960 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:33:52.512385 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506679 6960 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:33:52.512855 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506682 6960 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:33:52.512855 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506685 6960 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:33:52.512855 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.506687 6960 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:33:52.512855 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.506783 6960 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 18:33:52.512855 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.506798 6960 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 18:33:52.512855 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.506805 6960 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 18:33:52.512855 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.506809 6960 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 18:33:52.512855 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.506814 6960 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 18:33:52.512855 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.506817 6960 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 18:33:52.512855 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.506821 6960 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 18:33:52.512855 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.506826 6960 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 18:33:52.512855 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.506829 6960 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 18:33:52.512855 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.506832 6960 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 18:33:52.512855 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.506836 6960 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 18:33:52.512855 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.506839 6960 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 18:33:52.512855 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.506843 6960 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 18:33:52.512855 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.506846 6960 flags.go:64] FLAG: --cgroup-root="" Oct 30 18:33:52.512855 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.506849 6960 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 18:33:52.512855 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.506854 6960 flags.go:64] FLAG: --client-ca-file="" Oct 30 18:33:52.512855 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.506857 6960 flags.go:64] FLAG: --cloud-provider="external" Oct 30 18:33:52.512855 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.506860 6960 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 18:33:52.512855 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.506864 6960 flags.go:64] FLAG: --cluster-domain="" Oct 30 18:33:52.512855 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.506866 6960 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 18:33:52.513462 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.506870 6960 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 18:33:52.513462 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.506873 6960 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 18:33:52.513462 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.506876 6960 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 18:33:52.513462 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.506880 6960 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 18:33:52.513462 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.506883 6960 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 18:33:52.513462 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.506886 6960 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 18:33:52.513462 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.506889 6960 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 18:33:52.513462 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.506892 6960 flags.go:64] FLAG: --contention-profiling="false" Oct 30 18:33:52.513462 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.506895 6960 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 18:33:52.513462 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.506898 6960 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 18:33:52.513462 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.506901 6960 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 18:33:52.513462 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.506904 6960 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 18:33:52.513462 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.506908 6960 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 18:33:52.513462 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.506911 6960 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 18:33:52.513462 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.506914 6960 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 18:33:52.513462 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.506917 6960 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 18:33:52.513462 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.506921 6960 flags.go:64] FLAG: --enable-server="true" Oct 30 18:33:52.513462 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.506923 6960 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 18:33:52.513462 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.506928 6960 flags.go:64] FLAG: --event-burst="100" Oct 30 18:33:52.513462 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.506931 6960 flags.go:64] FLAG: --event-qps="50" Oct 30 18:33:52.513462 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.506934 6960 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 18:33:52.513462 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.506937 6960 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 18:33:52.513462 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.506940 6960 flags.go:64] FLAG: --eviction-hard="" Oct 30 18:33:52.513462 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.506944 6960 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 18:33:52.514030 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.506947 6960 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 18:33:52.514030 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.506950 6960 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 18:33:52.514030 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.506953 6960 flags.go:64] FLAG: --eviction-soft="" Oct 30 18:33:52.514030 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.506958 6960 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 18:33:52.514030 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.506961 6960 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 18:33:52.514030 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.506963 6960 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 18:33:52.514030 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.506966 6960 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 18:33:52.514030 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.506969 6960 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 18:33:52.514030 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.506972 6960 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 18:33:52.514030 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.506975 6960 flags.go:64] FLAG: --feature-gates="" Oct 30 18:33:52.514030 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.506979 6960 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 18:33:52.514030 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.506983 6960 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 18:33:52.514030 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.506986 6960 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 18:33:52.514030 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.506989 6960 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 18:33:52.514030 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.506992 6960 flags.go:64] FLAG: --healthz-port="10248" Oct 30 18:33:52.514030 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.506995 6960 flags.go:64] FLAG: --help="false" Oct 30 18:33:52.514030 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.506998 6960 flags.go:64] FLAG: --hostname-override="ip-10-0-140-163.us-east-2.compute.internal" Oct 30 18:33:52.514030 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.507001 6960 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 18:33:52.514030 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.507005 6960 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 18:33:52.514030 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.507008 6960 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 18:33:52.514030 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.507011 6960 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 18:33:52.514030 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.507015 6960 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 18:33:52.514030 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.507019 6960 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 18:33:52.514587 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.507022 6960 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 18:33:52.514587 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.507025 6960 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 18:33:52.514587 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.507028 6960 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 18:33:52.514587 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.507031 6960 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 18:33:52.514587 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.507034 6960 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 18:33:52.514587 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.507037 6960 flags.go:64] FLAG: --kube-reserved="" Oct 30 18:33:52.514587 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.507040 6960 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 18:33:52.514587 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.507043 6960 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 18:33:52.514587 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.507046 6960 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 18:33:52.514587 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.507049 6960 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 18:33:52.514587 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.507052 6960 flags.go:64] FLAG: --lock-file="" Oct 30 18:33:52.514587 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.507055 6960 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 18:33:52.514587 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.507059 6960 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 18:33:52.514587 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.507062 6960 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 18:33:52.514587 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.507067 6960 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 18:33:52.514587 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.507070 6960 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 18:33:52.514587 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.507073 6960 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 18:33:52.514587 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.507076 6960 flags.go:64] FLAG: --logging-format="text" Oct 30 18:33:52.514587 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.507079 6960 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 18:33:52.514587 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.507082 6960 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 18:33:52.514587 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.507085 6960 flags.go:64] FLAG: --manifest-url="" Oct 30 18:33:52.514587 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.507089 6960 flags.go:64] FLAG: --manifest-url-header="" Oct 30 18:33:52.514587 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.507093 6960 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 18:33:52.514587 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.507096 6960 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 18:33:52.514587 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.507100 6960 flags.go:64] FLAG: --max-pods="110" Oct 30 18:33:52.515237 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.507103 6960 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 18:33:52.515237 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.507105 6960 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 18:33:52.515237 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.507108 6960 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 18:33:52.515237 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.507111 6960 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 18:33:52.515237 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.507114 6960 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 18:33:52.515237 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.507117 6960 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 18:33:52.515237 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.507122 6960 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 18:33:52.515237 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.507129 6960 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 18:33:52.515237 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.507132 6960 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 18:33:52.515237 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.507135 6960 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 18:33:52.515237 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.507138 6960 flags.go:64] FLAG: --pod-cidr="" Oct 30 18:33:52.515237 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.507156 6960 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 18:33:52.515237 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.507160 6960 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 18:33:52.515237 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.507165 6960 flags.go:64] FLAG: --pods-per-core="0" Oct 30 18:33:52.515237 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.507170 6960 flags.go:64] FLAG: --port="10250" Oct 30 18:33:52.515237 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.507174 6960 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 18:33:52.515237 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.507178 6960 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0a85cc7f9f2bfb89b" Oct 30 18:33:52.515237 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.507182 6960 flags.go:64] FLAG: --qos-reserved="" Oct 30 18:33:52.515237 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.507185 6960 flags.go:64] FLAG: --read-only-port="10255" Oct 30 18:33:52.515237 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.507189 6960 flags.go:64] FLAG: --register-node="true" Oct 30 18:33:52.515237 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.507193 6960 flags.go:64] FLAG: --register-with-taints="" Oct 30 18:33:52.515237 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.507196 6960 flags.go:64] FLAG: --registry-burst="10" Oct 30 18:33:52.515237 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.507199 6960 flags.go:64] FLAG: --registry-qps="5" Oct 30 18:33:52.515237 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.507202 6960 flags.go:64] FLAG: --reserved-cpus="" Oct 30 18:33:52.515237 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.507206 6960 flags.go:64] FLAG: --reserved-memory="" Oct 30 18:33:52.515237 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.507210 6960 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 18:33:52.515854 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.507213 6960 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 18:33:52.515854 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.507216 6960 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 18:33:52.515854 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.507219 6960 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 18:33:52.515854 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.507221 6960 flags.go:64] FLAG: --runonce="false" Oct 30 18:33:52.515854 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.507224 6960 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 18:33:52.515854 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.507228 6960 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 18:33:52.515854 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.507231 6960 flags.go:64] FLAG: --seccomp-default="false" Oct 30 18:33:52.515854 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.507233 6960 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 18:33:52.515854 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.507236 6960 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 18:33:52.515854 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.507239 6960 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 18:33:52.515854 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.507242 6960 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 18:33:52.515854 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.507245 6960 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 18:33:52.515854 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.507249 6960 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 18:33:52.515854 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.507252 6960 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 18:33:52.515854 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.507255 6960 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 18:33:52.515854 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.507258 6960 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 18:33:52.515854 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.507261 6960 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 18:33:52.515854 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.507264 6960 flags.go:64] FLAG: --system-cgroups="" Oct 30 18:33:52.515854 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.507267 6960 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 18:33:52.515854 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.507272 6960 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 18:33:52.515854 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.507275 6960 flags.go:64] FLAG: --tls-cert-file="" Oct 30 18:33:52.515854 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.507278 6960 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 18:33:52.515854 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.507282 6960 flags.go:64] FLAG: --tls-min-version="" Oct 30 18:33:52.515854 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.507286 6960 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 18:33:52.515854 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.507292 6960 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 18:33:52.516483 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.507299 6960 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 18:33:52.516483 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.507303 6960 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 18:33:52.516483 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.507307 6960 flags.go:64] FLAG: --v="2" Oct 30 18:33:52.516483 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.507314 6960 flags.go:64] FLAG: --version="false" Oct 30 18:33:52.516483 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.507320 6960 flags.go:64] FLAG: --vmodule="" Oct 30 18:33:52.516483 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.507326 6960 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 18:33:52.516483 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.507330 6960 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 18:33:52.516483 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507476 6960 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:33:52.516483 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507480 6960 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:33:52.516483 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507483 6960 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:33:52.516483 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507486 6960 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:33:52.516483 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507488 6960 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:33:52.516483 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507491 6960 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:33:52.516483 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507494 6960 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:33:52.516483 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507497 6960 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:33:52.516483 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507500 6960 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:33:52.516483 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507502 6960 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:33:52.516483 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507505 6960 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:33:52.516483 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507508 6960 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:33:52.516483 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507513 6960 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:33:52.516483 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507515 6960 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:33:52.516483 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507518 6960 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:33:52.517030 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507521 6960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:33:52.517030 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507523 6960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:33:52.517030 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507526 6960 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:33:52.517030 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507528 6960 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:33:52.517030 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507531 6960 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:33:52.517030 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507533 6960 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:33:52.517030 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507537 6960 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:33:52.517030 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507539 6960 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:33:52.517030 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507542 6960 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:33:52.517030 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507545 6960 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:33:52.517030 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507548 6960 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:33:52.517030 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507551 6960 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:33:52.517030 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507554 6960 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:33:52.517030 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507557 6960 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:33:52.517030 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507559 6960 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:33:52.517030 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507562 6960 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:33:52.517030 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507564 6960 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:33:52.517030 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507567 6960 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:33:52.517030 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507570 6960 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:33:52.517030 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507572 6960 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:33:52.517626 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507574 6960 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:33:52.517626 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507577 6960 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:33:52.517626 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507581 6960 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:33:52.517626 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507584 6960 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:33:52.517626 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507587 6960 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:33:52.517626 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507590 6960 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:33:52.517626 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507593 6960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:33:52.517626 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507595 6960 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:33:52.517626 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507598 6960 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:33:52.517626 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507602 6960 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:33:52.517626 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507604 6960 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:33:52.517626 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507607 6960 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:33:52.517626 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507609 6960 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:33:52.517626 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507612 6960 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:33:52.517626 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507615 6960 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:33:52.517626 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507617 6960 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:33:52.517626 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507619 6960 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:33:52.517626 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507624 6960 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:33:52.517626 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507627 6960 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:33:52.518172 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507630 6960 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:33:52.518172 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507633 6960 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:33:52.518172 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507635 6960 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:33:52.518172 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507638 6960 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:33:52.518172 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507641 6960 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:33:52.518172 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507643 6960 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:33:52.518172 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507646 6960 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:33:52.518172 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507648 6960 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:33:52.518172 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507651 6960 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:33:52.518172 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507653 6960 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:33:52.518172 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507656 6960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:33:52.518172 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507659 6960 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:33:52.518172 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507661 6960 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:33:52.518172 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507664 6960 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:33:52.518172 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507666 6960 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:33:52.518172 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507669 6960 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:33:52.518172 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507671 6960 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:33:52.518172 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507674 6960 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:33:52.518172 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507676 6960 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:33:52.518663 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507679 6960 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:33:52.518663 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507681 6960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:33:52.518663 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507684 6960 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:33:52.518663 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507687 6960 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:33:52.518663 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507690 6960 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:33:52.518663 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507692 6960 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:33:52.518663 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507695 6960 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:33:52.518663 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507698 6960 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:33:52.518663 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507701 6960 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:33:52.518663 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507703 6960 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:33:52.518663 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507706 6960 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:33:52.518663 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507709 6960 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:33:52.518663 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507711 6960 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:33:52.518663 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507714 6960 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:33:52.518663 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507717 6960 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:33:52.518663 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507719 6960 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:33:52.518663 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507722 6960 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:33:52.518663 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507724 6960 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:33:52.518663 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507727 6960 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:33:52.519172 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507730 6960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:33:52.519172 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507732 6960 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:33:52.519172 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507735 6960 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:33:52.519172 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507737 6960 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:33:52.519172 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507740 6960 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:33:52.519172 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507742 6960 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:33:52.519172 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507745 6960 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:33:52.519172 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507748 6960 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:33:52.519172 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507750 6960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:33:52.519172 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507753 6960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:33:52.519172 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507755 6960 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:33:52.519172 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507758 6960 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:33:52.519172 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507760 6960 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:33:52.519172 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507763 6960 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:33:52.519172 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.507788 6960 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 18:33:52.519740 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507828 6960 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:33:52.519740 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507834 6960 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:33:52.519740 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507837 6960 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:33:52.519740 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507840 6960 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:33:52.519740 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507843 6960 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:33:52.519740 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507845 6960 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:33:52.519740 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507848 6960 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:33:52.519740 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507850 6960 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:33:52.519740 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507853 6960 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:33:52.519740 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507857 6960 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:33:52.519740 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507861 6960 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:33:52.519740 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507864 6960 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:33:52.519740 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507867 6960 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:33:52.519740 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507869 6960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:33:52.519740 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507872 6960 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:33:52.519740 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507875 6960 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:33:52.519740 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507877 6960 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:33:52.519740 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507880 6960 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:33:52.519740 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507883 6960 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:33:52.519740 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507885 6960 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:33:52.520270 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507888 6960 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:33:52.520270 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507890 6960 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:33:52.520270 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507893 6960 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:33:52.520270 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507895 6960 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:33:52.520270 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507899 6960 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:33:52.520270 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507904 6960 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:33:52.520270 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507907 6960 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:33:52.520270 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507910 6960 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:33:52.520270 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507912 6960 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:33:52.520270 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507915 6960 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:33:52.520270 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507917 6960 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:33:52.520270 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507920 6960 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:33:52.520270 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507923 6960 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:33:52.520270 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507927 6960 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:33:52.520270 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507930 6960 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:33:52.520270 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507932 6960 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:33:52.520270 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507935 6960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:33:52.520270 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507937 6960 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:33:52.520270 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507940 6960 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:33:52.520722 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507943 6960 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:33:52.520722 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507945 6960 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:33:52.520722 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507948 6960 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:33:52.520722 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507950 6960 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:33:52.520722 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507953 6960 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:33:52.520722 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507955 6960 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:33:52.520722 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507958 6960 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:33:52.520722 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507961 6960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:33:52.520722 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507963 6960 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:33:52.520722 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507965 6960 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:33:52.520722 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507968 6960 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:33:52.520722 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507971 6960 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:33:52.520722 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507974 6960 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:33:52.520722 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507977 6960 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:33:52.520722 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507980 6960 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:33:52.520722 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507982 6960 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:33:52.520722 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507986 6960 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:33:52.520722 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507988 6960 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:33:52.520722 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507992 6960 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:33:52.521354 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507995 6960 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:33:52.521354 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.507997 6960 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:33:52.521354 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.508000 6960 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:33:52.521354 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.508002 6960 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:33:52.521354 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.508005 6960 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:33:52.521354 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.508007 6960 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:33:52.521354 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.508010 6960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:33:52.521354 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.508014 6960 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:33:52.521354 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.508016 6960 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:33:52.521354 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.508019 6960 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:33:52.521354 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.508022 6960 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:33:52.521354 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.508024 6960 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:33:52.521354 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.508027 6960 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:33:52.521354 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.508030 6960 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:33:52.521354 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.508032 6960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:33:52.521354 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.508035 6960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:33:52.521354 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.508037 6960 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:33:52.521354 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.508040 6960 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:33:52.521354 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.508043 6960 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:33:52.521867 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.508045 6960 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:33:52.521867 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.508048 6960 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:33:52.521867 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.508051 6960 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:33:52.521867 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.508053 6960 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:33:52.521867 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.508056 6960 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:33:52.521867 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.508058 6960 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:33:52.521867 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.508061 6960 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:33:52.521867 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.508064 6960 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:33:52.521867 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.508067 6960 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:33:52.521867 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.508069 6960 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:33:52.521867 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.508072 6960 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:33:52.521867 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.508075 6960 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:33:52.521867 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.508079 6960 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:33:52.521867 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.508081 6960 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:33:52.521867 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.508084 6960 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:33:52.521867 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.508086 6960 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:33:52.521867 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.508089 6960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:33:52.521867 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.508091 6960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:33:52.521867 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.508094 6960 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:33:52.521867 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.508097 6960 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:33:52.522417 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.508100 6960 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:33:52.522417 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.508103 6960 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:33:52.522417 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.508106 6960 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:33:52.522417 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.508109 6960 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:33:52.522417 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.508112 6960 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:33:52.522417 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.508114 6960 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:33:52.522417 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.508117 6960 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:33:52.522417 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.508119 6960 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:33:52.522417 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.508122 6960 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:33:52.522417 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.513032 6960 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 18:33:52.522417 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.513048 6960 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 18:33:52.522417 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.513071 6960 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 18:33:52.522417 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.513076 6960 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 18:33:52.522417 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513136 6960 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:33:52.522417 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513162 6960 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:33:52.522417 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513166 6960 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:33:52.522417 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513169 6960 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:33:52.522417 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513172 6960 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:33:52.522417 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513175 6960 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:33:52.522417 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513177 6960 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:33:52.522920 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513180 6960 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:33:52.522920 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513183 6960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:33:52.522920 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513186 6960 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:33:52.522920 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513189 6960 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:33:52.522920 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513191 6960 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:33:52.522920 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513194 6960 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:33:52.522920 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513197 6960 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:33:52.522920 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513200 6960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:33:52.522920 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513202 6960 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:33:52.522920 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513206 6960 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:33:52.522920 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513211 6960 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:33:52.522920 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513214 6960 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:33:52.522920 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513218 6960 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:33:52.522920 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513220 6960 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:33:52.522920 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513223 6960 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:33:52.522920 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513227 6960 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:33:52.522920 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513230 6960 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:33:52.522920 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513233 6960 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:33:52.522920 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513235 6960 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:33:52.523402 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513238 6960 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:33:52.523402 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513241 6960 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:33:52.523402 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513244 6960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:33:52.523402 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513247 6960 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:33:52.523402 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513250 6960 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:33:52.523402 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513252 6960 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:33:52.523402 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513255 6960 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:33:52.523402 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513258 6960 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:33:52.523402 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513261 6960 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:33:52.523402 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513264 6960 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:33:52.523402 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513267 6960 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:33:52.523402 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513269 6960 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:33:52.523402 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513272 6960 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:33:52.523402 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513274 6960 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:33:52.523402 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513277 6960 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:33:52.523402 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513280 6960 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:33:52.523402 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513283 6960 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:33:52.523402 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513285 6960 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:33:52.523402 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513288 6960 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:33:52.523844 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513290 6960 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:33:52.523844 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513304 6960 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:33:52.523844 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513307 6960 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:33:52.523844 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513310 6960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:33:52.523844 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513312 6960 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:33:52.523844 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513315 6960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:33:52.523844 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513317 6960 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:33:52.523844 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513320 6960 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:33:52.523844 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513323 6960 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:33:52.523844 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513326 6960 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:33:52.523844 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513329 6960 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:33:52.523844 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513332 6960 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:33:52.523844 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513334 6960 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:33:52.523844 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513337 6960 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:33:52.523844 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513339 6960 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:33:52.523844 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513342 6960 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:33:52.523844 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513345 6960 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:33:52.523844 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513347 6960 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:33:52.523844 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513350 6960 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:33:52.523844 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513353 6960 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:33:52.524341 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513356 6960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:33:52.524341 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513358 6960 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:33:52.524341 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513361 6960 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:33:52.524341 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513363 6960 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:33:52.524341 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513366 6960 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:33:52.524341 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513369 6960 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:33:52.524341 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513371 6960 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:33:52.524341 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513374 6960 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:33:52.524341 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513377 6960 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:33:52.524341 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513379 6960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:33:52.524341 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513382 6960 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:33:52.524341 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513385 6960 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:33:52.524341 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513387 6960 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:33:52.524341 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513390 6960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:33:52.524341 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513399 6960 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:33:52.524341 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513402 6960 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:33:52.524341 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513405 6960 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:33:52.524341 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513407 6960 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:33:52.524341 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513410 6960 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:33:52.525195 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513413 6960 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:33:52.525195 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513415 6960 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:33:52.525195 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513418 6960 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:33:52.525195 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513421 6960 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:33:52.525195 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513423 6960 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:33:52.525195 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513426 6960 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:33:52.525195 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513429 6960 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:33:52.525195 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513431 6960 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:33:52.525195 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513435 6960 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:33:52.525195 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513439 6960 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:33:52.525195 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513442 6960 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:33:52.525195 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513444 6960 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:33:52.525195 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513447 6960 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:33:52.525195 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513450 6960 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:33:52.525195 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513452 6960 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:33:52.525195 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513455 6960 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:33:52.525195 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513457 6960 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:33:52.525195 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513460 6960 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:33:52.525195 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513463 6960 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:33:52.525195 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513465 6960 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:33:52.526000 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513468 6960 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:33:52.526000 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513471 6960 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:33:52.526000 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.513499 6960 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 18:33:52.526000 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513633 6960 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:33:52.526000 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513638 6960 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:33:52.526000 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513642 6960 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:33:52.526000 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513646 6960 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:33:52.526000 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513650 6960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:33:52.526000 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513655 6960 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:33:52.526000 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513659 6960 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:33:52.526000 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513663 6960 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:33:52.526000 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513666 6960 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:33:52.526000 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513669 6960 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:33:52.526000 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513672 6960 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:33:52.526000 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513675 6960 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:33:52.526402 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513678 6960 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:33:52.526402 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513680 6960 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:33:52.526402 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513683 6960 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:33:52.526402 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513686 6960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:33:52.526402 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513688 6960 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:33:52.526402 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513691 6960 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:33:52.526402 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513693 6960 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:33:52.526402 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513696 6960 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:33:52.526402 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513699 6960 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:33:52.526402 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513701 6960 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:33:52.526402 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513704 6960 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:33:52.526402 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513707 6960 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:33:52.526402 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513709 6960 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:33:52.526402 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513712 6960 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:33:52.526402 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513715 6960 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:33:52.526402 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513717 6960 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:33:52.526402 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513720 6960 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:33:52.526402 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513722 6960 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:33:52.526402 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513725 6960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:33:52.526402 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513728 6960 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:33:52.526935 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513730 6960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:33:52.526935 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513733 6960 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:33:52.526935 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513735 6960 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:33:52.526935 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513738 6960 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:33:52.526935 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513740 6960 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:33:52.526935 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513747 6960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:33:52.526935 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513751 6960 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:33:52.526935 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513754 6960 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:33:52.526935 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513756 6960 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:33:52.526935 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513759 6960 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:33:52.526935 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513762 6960 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:33:52.526935 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513765 6960 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:33:52.526935 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513767 6960 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:33:52.526935 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513770 6960 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:33:52.526935 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513772 6960 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:33:52.526935 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513775 6960 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:33:52.526935 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513778 6960 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:33:52.526935 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513780 6960 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:33:52.526935 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513783 6960 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:33:52.526935 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513785 6960 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:33:52.527498 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513788 6960 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:33:52.527498 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513790 6960 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:33:52.527498 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513793 6960 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:33:52.527498 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513795 6960 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:33:52.527498 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513798 6960 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:33:52.527498 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513800 6960 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:33:52.527498 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513803 6960 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:33:52.527498 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513806 6960 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:33:52.527498 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513809 6960 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:33:52.527498 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513811 6960 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:33:52.527498 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513814 6960 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:33:52.527498 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513816 6960 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:33:52.527498 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513819 6960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:33:52.527498 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513821 6960 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:33:52.527498 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513824 6960 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:33:52.527498 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513826 6960 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:33:52.527498 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513829 6960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:33:52.527498 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513833 6960 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:33:52.527498 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513835 6960 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:33:52.527498 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513839 6960 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:33:52.527978 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513841 6960 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:33:52.527978 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513844 6960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:33:52.527978 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513847 6960 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:33:52.527978 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513849 6960 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:33:52.527978 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513852 6960 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:33:52.527978 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513855 6960 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:33:52.527978 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513857 6960 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:33:52.527978 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513860 6960 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:33:52.527978 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513862 6960 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:33:52.527978 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513865 6960 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:33:52.527978 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513867 6960 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:33:52.527978 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513870 6960 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:33:52.527978 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513873 6960 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:33:52.527978 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513875 6960 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:33:52.527978 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513878 6960 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:33:52.527978 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513881 6960 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:33:52.527978 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513883 6960 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:33:52.527978 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513885 6960 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:33:52.527978 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513888 6960 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:33:52.527978 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513891 6960 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:33:52.528508 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513894 6960 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:33:52.528508 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513896 6960 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:33:52.528508 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513899 6960 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:33:52.528508 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513901 6960 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:33:52.528508 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513904 6960 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:33:52.528508 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513907 6960 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:33:52.528508 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513909 6960 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:33:52.528508 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513912 6960 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:33:52.528508 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513915 6960 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:33:52.528508 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513919 6960 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:33:52.528508 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513922 6960 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:33:52.528508 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513924 6960 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:33:52.528508 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513927 6960 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:33:52.528508 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513930 6960 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:33:52.528508 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.513954 6960 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 18:33:52.528508 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513991 6960 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:33:52.528944 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513996 6960 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:33:52.528944 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.513999 6960 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:33:52.528944 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514002 6960 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:33:52.528944 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514006 6960 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:33:52.528944 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514009 6960 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:33:52.528944 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514011 6960 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:33:52.528944 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514014 6960 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:33:52.528944 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514016 6960 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:33:52.528944 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514019 6960 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:33:52.528944 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514022 6960 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:33:52.528944 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514024 6960 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:33:52.528944 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514027 6960 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:33:52.528944 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514030 6960 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:33:52.528944 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514033 6960 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:33:52.528944 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514036 6960 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:33:52.528944 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514039 6960 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:33:52.528944 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514041 6960 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:33:52.528944 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514044 6960 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:33:52.528944 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514047 6960 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:33:52.529432 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514049 6960 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:33:52.529432 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514052 6960 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:33:52.529432 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514055 6960 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:33:52.529432 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514057 6960 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:33:52.529432 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514060 6960 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:33:52.529432 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514062 6960 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:33:52.529432 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514065 6960 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:33:52.529432 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514069 6960 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:33:52.529432 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514073 6960 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:33:52.529432 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514076 6960 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:33:52.529432 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514080 6960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:33:52.529432 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514085 6960 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:33:52.529432 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514089 6960 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:33:52.529432 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514092 6960 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:33:52.529432 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514095 6960 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:33:52.529432 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514098 6960 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:33:52.529432 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514100 6960 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:33:52.529432 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514103 6960 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:33:52.529432 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514105 6960 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:33:52.529892 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514108 6960 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:33:52.529892 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514110 6960 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:33:52.529892 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514114 6960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:33:52.529892 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514116 6960 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:33:52.529892 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514119 6960 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:33:52.529892 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514121 6960 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:33:52.529892 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514124 6960 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:33:52.529892 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514126 6960 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:33:52.529892 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514129 6960 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:33:52.529892 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514132 6960 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:33:52.529892 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514134 6960 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:33:52.529892 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514137 6960 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:33:52.529892 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514156 6960 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:33:52.529892 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514160 6960 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:33:52.529892 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514162 6960 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:33:52.529892 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514165 6960 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:33:52.529892 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514167 6960 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:33:52.529892 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514170 6960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:33:52.529892 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514172 6960 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:33:52.529892 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514175 6960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:33:52.530441 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514178 6960 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:33:52.530441 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514180 6960 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:33:52.530441 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514183 6960 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:33:52.530441 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514185 6960 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:33:52.530441 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514190 6960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:33:52.530441 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514193 6960 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:33:52.530441 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514196 6960 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:33:52.530441 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514198 6960 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:33:52.530441 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514201 6960 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:33:52.530441 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514204 6960 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:33:52.530441 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514206 6960 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:33:52.530441 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514209 6960 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:33:52.530441 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514211 6960 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:33:52.530441 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514214 6960 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:33:52.530441 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514217 6960 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:33:52.530441 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514219 6960 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:33:52.530441 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514222 6960 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:33:52.530441 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514224 6960 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:33:52.530441 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514227 6960 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:33:52.530441 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514229 6960 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:33:52.530932 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514231 6960 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:33:52.530932 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514234 6960 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:33:52.530932 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514236 6960 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:33:52.530932 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514239 6960 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:33:52.530932 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514242 6960 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:33:52.530932 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514245 6960 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:33:52.530932 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514247 6960 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:33:52.530932 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514250 6960 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:33:52.530932 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514252 6960 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:33:52.530932 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514255 6960 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:33:52.530932 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514257 6960 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:33:52.530932 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514260 6960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:33:52.530932 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514262 6960 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:33:52.530932 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514266 6960 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:33:52.530932 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514268 6960 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:33:52.530932 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514271 6960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:33:52.530932 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514276 6960 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:33:52.530932 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514279 6960 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:33:52.530932 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514281 6960 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:33:52.530932 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514284 6960 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:33:52.531510 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514286 6960 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:33:52.531510 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514289 6960 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:33:52.531510 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514291 6960 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:33:52.531510 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514294 6960 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:33:52.531510 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514296 6960 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:33:52.531510 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514299 6960 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:33:52.531510 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:33:52.514302 6960 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:33:52.531510 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.514354 6960 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 18:33:52.531510 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.515387 6960 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 18:33:52.531510 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.515439 6960 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 18:33:52.531510 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.515619 6960 server.go:1014] "Starting client certificate rotation" Oct 30 18:33:52.531510 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.515762 6960 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 18:33:52.531510 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.515823 6960 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 18:27:43 +0000 UTC" deadline="2028-05-30 17:20:24.041893201 +0000 UTC" Oct 30 18:33:52.531510 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.515867 6960 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13870h46m31.526030141s" Oct 30 18:33:52.531510 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.516132 6960 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 18:33:52.531510 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.517179 6960 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 18:33:52.531900 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.518682 6960 log.go:25] "Validated CRI v1 runtime API" Oct 30 18:33:52.531900 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.520704 6960 log.go:25] "Validated CRI v1 image API" Oct 30 18:33:52.531900 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.521526 6960 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 18:33:52.531900 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.524121 6960 fs.go:135] Filesystem UUIDs: map[5faaf9ec-896a-4a34-b4bc-b7d4e78ea6b0:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 e22d9d11-52dc-4fb5-a2f5-fcb44834329f:/dev/nvme0n1p4] Oct 30 18:33:52.532037 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.524170 6960 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/06687992306a30ad3c6f9044932282015ccabc988888a1c185e89fa381425904/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/06687992306a30ad3c6f9044932282015ccabc988888a1c185e89fa381425904/userdata/shm major:0 minor:480 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/095a58f8a000a16f80609d89f8adf94a3432280b70698e057daf05d423517c6f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/095a58f8a000a16f80609d89f8adf94a3432280b70698e057daf05d423517c6f/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/119b97a32ab1fa4705dfa33313f5a74d8b7b63668d39e87f373a39bbe1dd070e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/119b97a32ab1fa4705dfa33313f5a74d8b7b63668d39e87f373a39bbe1dd070e/userdata/shm major:0 minor:334 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/172feea2873431f0fbd887a83aadcc45053c61f7631d004fbfbf5d3f0aa4dd77/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/172feea2873431f0fbd887a83aadcc45053c61f7631d004fbfbf5d3f0aa4dd77/userdata/shm major:0 minor:64 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2c0ae33ab021d9f5442b117896d2dab11ba19c65301da78cb6bad9ac5d0117dc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2c0ae33ab021d9f5442b117896d2dab11ba19c65301da78cb6bad9ac5d0117dc/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2df7bb0c79d3346dc1213c978fbd1bf6cf2feee082618ee2378c05df610217e3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2df7bb0c79d3346dc1213c978fbd1bf6cf2feee082618ee2378c05df610217e3/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/32522431fb329153a6eeebccc7680bd24dcdea651248be4cc3d5f50b8bb5a1e3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/32522431fb329153a6eeebccc7680bd24dcdea651248be4cc3d5f50b8bb5a1e3/userdata/shm major:0 minor:434 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/37d5c880b26121a7ab8d7ad2034e9521e55fd3ef0e30c419741bfabac5b7882a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/37d5c880b26121a7ab8d7ad2034e9521e55fd3ef0e30c419741bfabac5b7882a/userdata/shm major:0 minor:364 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3fc6a6002c134d13eff65b44fca6ed477f75262fb065531496148ca5cc8559e7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3fc6a6002c134d13eff65b44fca6ed477f75262fb065531496148ca5cc8559e7/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/50b0bb5c3b75d6ed563d7acd7bc142c8e6e66f899b26bcdb63b7e097dea2f1fa/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/50b0bb5c3b75d6ed563d7acd7bc142c8e6e66f899b26bcdb63b7e097dea2f1fa/userdata/shm major:0 minor:304 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/56cf64efd808a743e34aa2da5f607cdce99f15da9da3be52befbd6612366c7d4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/56cf64efd808a743e34aa2da5f607cdce99f15da9da3be52befbd6612366c7d4/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/683740f7ded55252e4418cbf3eef7b202800bc07ad67aee5e15472576a901d63/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/683740f7ded55252e4418cbf3eef7b202800bc07ad67aee5e15472576a901d63/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6f84cca17983b8b70b45b2bcdc32e64617a0f4bb6889b0c8fa47567b6a7149eb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6f84cca17983b8b70b45b2bcdc32e64617a0f4bb6889b0c8fa47567b6a7149eb/userdata/shm major:0 minor:261 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8ae7cf2fa0bd11e84801e93bc28861efbd30ad203a94848ef0f5199e314f4f4c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8ae7cf2fa0bd11e84801e93bc28861efbd30ad203a94848ef0f5199e314f4f4c/userdata/shm major:0 minor:70 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/908390dea71dfceecda52d389bc886f06dadc8f7acb975b9402fe99fc931d605/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/908390dea71dfceecda52d389bc886f06dadc8f7acb975b9402fe99fc931d605/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a2a9935a2a2d2b39234f40a39264586ed51525af2ed3b73ad1b2f5be53e581a7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a2a9935a2a2d2b39234f40a39264586ed51525af2ed3b73ad1b2f5be53e581a7/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a3060082255cf6b51651149218af4bdbba6f1b0033acf76294ab9f552e69ec40/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a3060082255cf6b51651149218af4bdbba6f1b0033acf76294ab9f552e69ec40/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b40da5c1f34f4f94eb250f3bed5492801d825f9ce4f5fe472b32b5fae787bd59/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b40da5c1f34f4f94eb250f3bed5492801d825f9ce4f5fe472b32b5fae787bd59/userdata/shm major:0 minor:475 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c9ae698e2ddbafabe23051c119ab3b29049da90efcd4e436acdec3095cd3fa73/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c9ae698e2ddbafabe23051c119ab3b29049da90efcd4e436acdec3095cd3fa73/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cf60a7a7a606fdc05b761a1d4e2408b15e4ec237e5a5c22d1b6a49bde2a24ad5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cf60a7a7a606fdc05b761a1d4e2408b15e4ec237e5a5c22d1b6a49bde2a24ad5/userdata/shm major:0 minor:263 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ebb7a6cb2f181cb9daf6993af1f5e8a539dc87cc81cd568fb2eb857afff8935c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ebb7a6cb2f181cb9daf6993af1f5e8a539dc87cc81cd568fb2eb857afff8935c/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f4c144adff68eb30cf1418f5b0c943b73d737798daeb89a49a0e73db0db3b1db/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f4c144adff68eb30cf1418f5b0c943b73d737798daeb89a49a0e73db0db3b1db/userdata/shm major:0 minor:478 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f653dd2bb9642410489171060113e275c78337a3f25db513e81c1722e9966388/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f653dd2bb9642410489171060113e275c78337a3f25db513e81c1722e9966388/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f7b465a0d6ebcaf211c2d96f236da176e968470550f58af269fdd08aa306e298/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f7b465a0d6ebcaf211c2d96f236da176e968470550f58af269fdd08aa306e298/userdata/shm major:0 minor:359 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/05cad8b6-56f5-4bd1-b12d-b2ae456b7027/volumes/kubernetes.io~projected/kube-api-access-vj8gn:{mountpoint:/var/lib/kubelet/pods/05cad8b6-56f5-4bd1-b12d-b2ae456b7027/volumes/kubernetes.io~projected/kube-api-access-vj8gn major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0a1badb7-be53-4b7f-933f-23e2599ab5ba/volumes/kubernetes.io~projected/kube-api-access-5pf82:{mountpoint:/var/lib/kubelet/pods/0a1badb7-be53-4b7f-933f-23e2599ab5ba/volumes/kubernetes.io~projected/kube-api-access-5pf82 major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0a1badb7-be53-4b7f-933f-23e2599ab5ba/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/0a1badb7-be53-4b7f-933f-23e2599ab5ba/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/158becf0-d6cf-4917-8329-4f64ae5a4ca4/volumes/kubernetes.io~projected/kube-api-access-4zndl:{mountpoint:/var/lib/kubelet/pods/158becf0-d6cf-4917-8329-4f64ae5a4ca4/volumes/kubernetes.io~projected/kube-api-access-4zndl major:0 minor:333 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/158becf0-d6cf-4917-8329-4f64ae5a4ca4/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/158becf0-d6cf-4917-8329-4f64ae5a4ca4/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:328 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/158becf0-d6cf-4917-8329-4f64ae5a4ca4/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/158becf0-d6cf-4917-8329-4f64ae5a4ca4/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:332 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/158becf0-d6cf-4917-8329-4f64ae5a4ca4/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/158becf0-d6cf-4917-8329-4f64ae5a4ca4/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:327 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/158becf0-d6cf-4917-8329-4f64ae5a4ca4/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/158becf0-d6cf-4917-8329-4f64ae5a4ca4/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:329 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/158becf0-d6cf-4917-8329-4f64ae5a4ca4/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/158becf0-d6cf-4917-8329-4f64ae5a4ca4/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:331 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/158becf0-d6cf-4917-8329-4f64ae5a4ca4/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/158becf0-d6cf-4917-8329-4f64ae5a4ca4/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:330 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2069a85a-44e7-4c98-9f47-7fad4416c15b/volumes/kubernetes.io~projected/kube-api-access-wq2pp:{mountpoint:/var/lib/kubelet/pods/2069a85a-44e7-4c98-9f47-7fad4416c15b/volumes/kubernetes.io~projected/kube-api-access-wq2pp major:0 minor:341 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2069a85a-44e7-4c98-9f47-7fad4416c15b/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/2069a85a-44e7-4c98-9f47-7fad4416c15b/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:340 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2069a85a-44e7-4c98-9f47-7fad4416c15b/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/2069a85a-44e7-4c98-9f47-7fad4416c15b/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:339 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2069a85a-44e7-4c98-9f47-7fad4416c15b/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/2069a85a-44e7-4c98-9f47-7fad4416c15b/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:338 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/25377c15-6e29-46b3-9f1b-9b1fc38570f5/volumes/kubernetes.io~projected/kube-api-access-6546m:{mountpoint:/var/lib/kubelet/pods/25377c15-6e29-46b3-9f1b-9b1fc38570f5/volumes/kubernetes.io~projected/kube-api-access-6546m major:0 minor:361 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/25377c15-6e29-46b3-9f1b-9b1fc38570f5/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/25377c15-6e29-46b3-9f1b-9b1fc38570f5/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:355 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/312dae04-dfda-41da-90ca-36b3b44a12ea/volumes/kubernetes.io~projected/kube-api-access-z8422:{mountpoint:/var/lib/kubelet/pods/312dae04-dfda-41da-90ca-36b3b44a12ea/volumes/kubernetes.io~projected/kube-api-access-z8422 major:0 minor:257 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/312dae04-dfda-41da-90ca-36b3b44a12ea/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/312dae04-dfda-41da-90ca-36b3b44a12ea/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:250 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/312dae04-dfda-41da-90ca-36b3b44a12ea/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/312dae04-dfda-41da-90ca-36b3b44a12ea/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:251 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/433ed181-b45e-45c0-a593-9cb0b6c09348/volumes/kubernetes.io~projected/kube-api-access-g52xm:{mountpoint:/var/lib/kubelet/pods/433ed181-b45e-45c0-a593-9cb0b6c09348/volumes/kubernetes.io~projected/kube-api-access-g52xm major:0 minor:259 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/433ed181-b45e-45c0-a593-9cb0b6c09348/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/433ed181-b45e-45c0-a593-9cb0b6c09348/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:247 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/433ed181-b45e-45c0-a593-9cb0b6c09348/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/433ed181-b45e-45c0-a593-9cb0b6c09348/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:249 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4f8841c0-4cf3-45d9-9301-957d4fffe1ce/volumes/kubernetes.io~projected/kube-api-access-x9nwr:{mountpoint:/var/lib/kubelet/pods/4f8841c0-4cf3-45d9-9301-957d4fffe1ce/volumes/kubernetes.io~projected/kube-api-access-x9nwr major:0 minor:258 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4f8841c0-4cf3-45d9-9301-957d4fffe1ce/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/4f8841c0-4cf3-45d9-9301-957d4fffe1ce/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:248 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4f8841c0-4cf3-45d9-9301-957d4fffe1ce/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/4f8841c0-4cf3-45d9-9301-957d4fffe1ce/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:252 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5b826abe-3f79-43ab-8813-2406d5c4dae9/volumes/kubernetes.io~projected/kube-api-access-kxzzq:{mountpoint:/var/lib/kubelet/pods/5b826abe-3f79-43ab-8813-2406d5c4dae9/volumes/kubernetes.io~projected/kube-api-access-kxzzq major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6c1405e1-dd94-4151-9b4f-6392155a2023/volumes/kubernetes.io~projected/kube-api-access-6z8lm:{mountpoint:/var/lib/kubelet/pods/6c1405e1-dd94-4151-9b4f-6392155a2023/volumes/kubernetes.io~projected/kube-api-access-6z8lm major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6c1405e1-dd94-4151-9b4f-6392155a2023/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/6c1405e1-dd94-4151-9b4f-6392155a2023/volumes/kubernetes.io~secret/metrics-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:285 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d/volume-subpaths/web-config/alertmanager/9 major:0 minor:279 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d/volume-subpaths/web-config/config-reloader/6 major:0 minor:279 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:279 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d/volumes/kubernetes.io~empty-dir/config-out major:0 minor:290 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d/volumes/kubernetes.io~projected/kube-api-access-25trq:{mountpoint:/var/lib/kubelet/pods/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d/volumes/kubernetes.io~projected/kube-api-access-25trq major:0 minor:291 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d/volumes/kubernetes.io~projected/tls-assets major:0 minor:284 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:285 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d/volumes/kubernetes.io~secret/config-volume major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d/volumes/kubernetes.io~secret/web-config major:0 minor:279 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/81f52134-6a09-431f-9f80-412ae7bbead8/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/81f52134-6a09-431f-9f80-412ae7bbead8/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/81f52134-6a09-431f-9f80-412ae7bbead8/volumes/kubernetes.io~projected/kube-api-access-rdz5l:{mountpoint:/var/lib/kubelet/pods/81f52134-6a09-431f-9f80-412ae7bbead8/volumes/kubernetes.io~projected/kube-api-access-rdz5l major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/81f52134-6a09-431f-9f80-412ae7bbead8/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/81f52134-6a09-431f-9f80-412ae7bbead8/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8f76499e-97ad-4c03-8a72-a62f65619596/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/8f76499e-97ad-4c03-8a72-a62f65619596/volume-subpaths/web-config/config-reloader/2 major:0 minor:423 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8f76499e-97ad-4c03-8a72-a62f65619596/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/8f76499e-97ad-4c03-8a72-a62f65619596/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:423 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8f76499e-97ad-4c03-8a72-a62f65619596/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/8f76499e-97ad-4c03-8a72-a62f65619596/volume-subpaths/web-config/prometheus/15 major:0 minor:423 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8f76499e-97ad-4c03-8a72-a62f65619596/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/8f76499e-97ad-4c03-8a72-a62f65619596/volumes/kubernetes.io~empty-dir/config-out major:0 minor:429 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8f76499e-97ad-4c03-8a72-a62f65619596/volumes/kubernetes.io~projected/kube-api-access-jrzxm:{mountpoint:/var/lib/kubelet/pods/8f76499e-97ad-4c03-8a72-a62f65619596/volumes/kubernetes.io~projected/kube-api-access-jrzxm major:0 minor:416 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8f76499e-97ad-4c03-8a72-a62f65619596/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/8f76499e-97ad-4c03-8a72-a62f65619596/volumes/kubernetes.io~projected/tls-assets major:0 minor:427 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8f76499e-97ad-4c03-8a72-a62f65619596/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/8f76499e-97ad-4c03-8a72-a62f65619596/volumes/kubernetes.io~secret/config major:0 minor:431 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8f76499e-97ad-4c03-8a72-a62f65619596/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/8f76499e-97ad-4c03-8a72-a62f65619596/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:428 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8f76499e-97ad-4c03-8a72-a62f65619596/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/8f76499e-97ad-4c03-8a72-a62f65619596/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:422 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8f76499e-97ad-4c03-8a72-a62f65619596/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/8f76499e-97ad-4c03-8a72-a62f65619596/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:424 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8f76499e-97ad-4c03-8a72-a62f65619596/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/8f76499e-97ad-4c03-8a72-a62f65619596/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:430 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8f76499e-97ad-4c03-8a72-a62f65619596/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/8f76499e-97ad-4c03-8a72-a62f65619596/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:421 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8f76499e-97ad-4c03-8a72-a62f65619596/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/8f76499e-97ad-4c03-8a72-a62f65619596/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:426 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8f76499e-97ad-4c03-8a72-a62f65619596/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/8f76499e-97ad-4c03-8a72-a62f65619596/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:425 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8f76499e-97ad-4c03-8a72-a62f65619596/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/8f76499e-97ad-4c03-8a72-a62f65619596/volumes/kubernetes.io~secret/web-config major:0 minor:423 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/90488e60-cb70-4afa-a54b-6f333d345e84/volumes/kubernetes.io~projected/kube-api-access-xzfgl:{mountpoint:/var/lib/kubelet/pods/90488e60-cb70-4afa-a54b-6f333d345e84/volumes/kubernetes.io~projected/kube-api-access-xzfgl major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/90488e60-cb70-4afa-a54b-6f333d345e84/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/90488e60-cb70-4afa-a54b-6f333d345e84/volumes/kubernetes.io~secret/metrics-certs major:0 minor:474 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/91716cdd-99fb-4140-a52c-6d357ac74fb1/volumes/kubernetes.io~projected/kube-api-access-bbvzt:{mountpoint:/var/lib/kubelet/pods/91716cdd-99fb-4140-a52c-6d357ac74fb1/volumes/kubernetes.io~projected/kube-api-access-bbvzt major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/91716cdd-99fb-4140-a52c-6d357ac74fb1/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/91716cdd-99fb-4140-a52c-6d357ac74fb1/volumes/kubernetes.io~secret/cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b0f929fc-a9ea-4935-9fe7-770c5e7d4de5/volumes/kubernetes.io~projected/kube-api-access-4rncj:{mountpoint:/var/lib/kubelet/pods/b0f929fc-a9ea-4935-9fe7-770c5e7d4de5/volumes/kubernetes.io~projected/kube-api-access-4rncj major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b4961ad6-2fc3-4c01-a565-6f66068d2958/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/b4961ad6-2fc3-4c01-a565-6f66068d2958/volumes/kubernetes.io~secret/agent-certs major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d2baf406-5501-4717-b5e1-d145edfe9ddc/volumes/kubernetes.io~projected/kube-api-access-m4s85:{mountpoint:/var/lib/kubelet/pods/d2baf406-5501-4717-b5e1-d145edfe9ddc/volumes/kubernetes.io~projected/kube-api-access-m4s85 major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e163025a-b19f-44e2-9058-b65f31bfbcc4/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/e163025a-b19f-44e2-9058-b65f31bfbcc4/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:470 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e91ad8e0-05ce-420f-9594-445f4d48739c/volumes/kubernetes.io~projected/kube-api-access-nb2bf:{mountpoint:/var/lib/kubelet/pods/e91ad8e0-05ce-420f-9594-445f4d48739c/volumes/kubernetes.io~projected/kube-api-access-nb2bf major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eef45986-9475-4e3e-a3f9-fdb682474ff3/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/eef45986-9475-4e3e-a3f9-fdb682474ff3/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eef45986-9475-4e3e-a3f9-fdb682474ff3/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/eef45986-9475-4e3e-a3f9-fdb682474ff3/volumes/kubernetes.io~empty-dir/tmp major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eef45986-9475-4e3e-a3f9-fdb682474ff3/volumes/kubernetes.io~projected/kube-api-access-knxck:{mountpoint:/var/lib/kubelet/pods/eef45986-9475-4e3e-a3f9-fdb682474ff3/volumes/kubernetes.io~projected/kube-api-access-knxck major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ef86d9fa-a344-4886-b80c-89ce84f418ea/volumes/kubernetes.io~projected/kube-api-access-l927s:{mountpoint:/var/lib/kubelet/pods/ef86d9fa-a344-4886-b80c-89ce84f418ea/volumes/kubernetes.io~projected/kube-api-access-l927s major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fb66d280-cf24-4549-8fdb-89f16b6080c1/volumes/kubernetes.io~projected/kube-api-access-ztd9j:{mountpoint:/var/lib/kubelet/pods/fb66d280-cf24-4549-8fdb-89f16b6080c1/volumes/kubernetes.io~projected/kube-api-access-ztd9j major:0 minor:477 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/bb9f8aaec36e08d0185f4dfc6fc7588cde2991047c8da7dd3c8fe80b6df9de71/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/3f643b7b28f012fccae5bc4eed5ad35b5a81958f22a3a60521f2caff378474ce/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/0c10995a39a1419e8c42ab58a0ebe8e3b4f6862a791317b139836e013d5260a1/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/36a5599ac31b26f4fd0da220091e7f6bd44ecdd3e069e052fb28719b59c9e6af/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-111:{mountpoint:/var/lib/containers/storage/overlay/d787d78171db17aa00c87acd77065ccc7e4da380272b0000f1c0982445ebf37c/merged major:0 minor:111 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/3584b3d8d4f31dcb77504bb35283f52b46070af6038d01370c14e361fd5f6a34/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/c2fe8a4ee56015110a627eda5f575525dd8a9d5d4b80c908a64f50bea6611497/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/411b1143581ef4a115b2e3794f048478b22b1c678c5dd3823ceed72b9ae192e7/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/bbef020276965d0f61f5d13a493fa2229bbb5ebfe65c7986a4157baed9552387/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/246337a60b8f6198ee3a210e3ea4aac029ee741db1f92e6ea9b90b281e185c83/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/2f201ec6616917400a0cfe38c65587562617eb94fa26b62c3bc79a05bdd01444/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-140:{mountpoint:/var/lib/containers/storage/overlay/77a93aef571d0ab7f740c873fce8457c9ef4ec58397baffa1a834f5e58471881/merged major:0 minor:140 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/c9271e69be1f481c106dea0e1d5c63a21ff942af5184c9eb35c71801246d952c/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/0324e675c1e932d61f216114a21a4954007af9da79d52c259c9851f41ffa852e/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/d5ad784f8c579ff451745819f9d2274f2de9162a71085ff87cc46da94597b1d4/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/32d34c7a828d7dd468f2ed8665202c7061a1aa7c9e287c70035253c96ec36f0b/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/9681f3d361579a2b62240432bd99b79a32975794bbfdbbbd8910eb65be299b75/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/7a1192663b5062924aaff11a22bff761c9a6b61b67805ac2b07648896b8e5215/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/0aa12f74423fd91db4446fbc78ab76aa51da72e80744be006ea5811a29a31956/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/d158e518801de42ebf3a890cb163cbd35ab85189f029c75dc943ffacd2b7dbe3/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/b77f6b6b441ffb291a0b8dc505dad781a0581c24c90c8cfb6280554402a02e9b/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/4faeaba25714ae3e9e8cad19f4d65f5d405fd3a922a22706d88668b20990882c/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/bc7847d4e7426ac1ac101a964914cdcc1e2f33841e11f014994154485113565d/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/df33f8e2c89799c1adf6fd8fb6d305a2db4e6070d572bc6af340d4449f5cd608/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-216:{mountpoint:/var/lib/containers/storage/overlay/a6575f996a1634a78f3747839e5119cedce381badc1a0e75f3cb8c24c85c3c95/merged major:0 minor:216 fsType:overlay blockSize:0} overlay_0-218:{mountpoint:/var/lib/containers/storage/overlay/c223173c264e100691a61964f0c7e21b238ba507d0ffad78b5e2e1f49c3004a3/merged major:0 minor:218 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/07567cdc629b1c3ad4133d1566ba18d7625ee7beedefa429ecd4184f1f82f3ef/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/119b6b659c71a473d9915d8e484e7acda203b55252149e448dd92fee7e51ef97/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/9c35dace646d734b76aa73bba1768d39d77391d937adf76069f1cea21af0b28c/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/534f790050a6d1f3a2e5c332f5a03ea6ecdabd9cbf6ef7d3c06300016edf29e4/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-265:{mountpoint:/var/lib/containers/storage/overlay/f1b7d93a5b196f51bcf02afaf3fa8dcf62be1f69ed7640fbf64d5fb10df16ee8/merged major:0 minor:265 fsType:overlay blockSize:0} overlay_0-267:{mountpoint:/var/lib/containers/storage/overlay/5bb377fa3a4a21c030b29a8f6344b0f75e450344cc5e26a4a2ecb2958f1cc8db/merged major:0 minor:267 fsType:overlay blockSize:0} overlay_0-269:{mountpoint:/var/lib/containers/storage/overlay/90a69884e176888926edac2ca98394f4e844600cfd6ab729eeb26f8e3c111441/merged major:0 minor:269 fsType:overlay blockSize:0} overlay_0-271:{mountpoint:/var/lib/containers/storage/overlay/71592536ee5fa6473ee8e3416afed340a5ce4efbb7bdf330283cc86e4d460619/merged major:0 minor:271 fsType:overlay blockSize:0} overlay_0-277:{mountpoint:/var/lib/containers/storage/overlay/a1a4d330df18511eddc937581b37167cd2b094f0d93750b122f1989baf4131e5/merged major:0 minor:277 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/7debc4b412a5c61d67b23543dbde37437e09942924fe605ae6fe3f9effc96b20/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/e68fb9fc78e16fe4671ff136788076ffc6cfcd23c76921ceca214d618c7f01b0/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/2aa8fcd27f1ec53b89f1dd66379052eea6ecde7d7005e55f6c5cd6ae8e85b6d6/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-309:{mountpoint:/var/lib/containers/storage/overlay/fbd68d98a5216979c69c0716b827508250c1b389d19023e199065561c993ab84/merged major:0 minor:309 fsType:overlay blockSize:0} overlay_0-311:{mountpoint:/var/lib/containers/storage/overlay/d68c1a1662ce54438c2c9504bf5016a42273adb0baa979ed15b0a892a5ca26ca/merged major:0 minor:311 fsType:overlay blockSize:0} overlay_0-317:{mountpoint:/var/lib/containers/storage/overlay/087121c9433fa61a7856fda7593896149f17e9860ff6500b923adb978cd2d40d/merged major:0 minor:317 fsType:overlay blockSize:0} overlay_0-322:{mountpoint:/var/lib/containers/storage/overlay/e2b1c25536f523fd1cdbbfcc112c4bb8a97d7767fdf2c28b030e70b720b3054f/merged major:0 minor:322 fsType:overlay blockSize:0} overlay_0-336:{mountpoint:/var/lib/containers/storage/overlay/b83114fabc83ce14bb0bbedd8372c2d452c95ccfd02ff08d7b733d8a4db81409/merged major:0 minor:336 fsType:overlay blockSize:0} overlay_0-342:{mountpoint:/var/lib/containers/storage/overlay/e4a409f1619504684856ac4cba35d8cf98abfdf82b73a53e33425fa3bd383a3e/merged major:0 minor:342 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/c068921ee66f29fef5292fa3ad496428fe0f373143eb98c8e416e814f0b9f70c/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-350:{mountpoint:/var/lib/containers/storage/overlay/0da119d859f03c02b6ec338148aaba5e708ba73ae349104e63eebb4dcd3ab84f/merged major:0 minor:350 fsType:overlay blockSize:0} overlay_0-362:{mountpoint:/var/lib/containers/storage/overlay/425bd45d4cf3c65f8f4dabfb5cdcb6195f63d4819798e5d3dadb7d9e0027be46/merged major:0 minor:362 fsType:overlay blockSize:0} overlay_0-366:{mountpoint:/var/lib/containers/storage/overlay/a750d446d81e27cf368b7a70d16ea8457ddcd817cc6697238e8a69a2078d10c2/merged major:0 minor:366 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/51a7b60ce2248c18d1b1bb354ec5194977c06325b733e009879b98a8c0d5fdc3/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-374:{mountpoint:/var/lib/containers/storage/overlay/b10557f208360f5670bab044bdc80e234f6658ab61aed2c3fe795308163bbcd3/merged major:0 minor:374 fsType:overlay blockSize:0} overlay_0-379:{mountpoint:/var/lib/containers/storage/overlay/52249dad7f31ec39c03b6a51b524214eb1fd9eaabe5f7e26c7368c28702b2f6d/merged major:0 minor:379 fsType:overlay blockSize:0} overlay_0-381:{mountpoint:/var/lib/containers/storage/overlay/01bd4030476ad8da4f224d19eca9c8c3885b3b93f7ca6c8a1246b56501b4a59a/merged major:0 minor:381 fsType:overlay blockSize:0} overlay_0-386:{mountpoint:/var/lib/containers/storage/overlay/31dae3bb714e01d1d670053e3a8f52808a1b93877dbb2cd0079d65edc15375d6/merged major:0 minor:386 fsType:overlay blockSize:0} overlay_0-392:{mountpoint:/var/lib/containers/storage/overlay/6acfed9e58df29625aeb996d2028a55016c1bd76f4b5621d66fc9c6a91bdb1df/merged major:0 minor:392 fsType:overlay blockSize:0} overlay_0-397:{mountpoint:/var/lib/containers/storage/overlay/7884564cd2bed71fe34f5f1ca8a2a79cfeb97ee6d7c9eb1fc3d168b4d769d963/merged major:0 minor:397 fsType:overlay blockSize:0} overlay_0-400:{mountpoint:/var/lib/containers/storage/overlay/daf0197298fc61ee0532469136104af6ba2583ed5ada18a1189695d423e016cd/merged major:0 minor:400 fsType:overlay blockSize:0} overlay_0-403:{mountpoint:/var/lib/containers/storage/overlay/9ff1957fbbd4dc567e54786827da0edc7e850bcf517b9ba5865ef33a135526cd/merged major:0 minor:403 fsType:overlay blockSize:0} overlay_0-411:{mountpoint:/var/lib/containers/storage/overlay/8c73c97e7a028a90e2018edc5c6ed15c0cc60464f264e51687769bcbcb367bea/merged major:0 minor:411 fsType:overlay blockSize:0} overlay_0-432:{mountpoint:/var/lib/containers/storage/overlay/bb15614cc850f66b4580fe96d5b0f105b6a8c346fe09606b69ff0c7bcc936b2f/merged major:0 minor:432 fsType:overlay blockSize:0} overlay_0-440:{mountpoint:/var/lib/containers/storage/overlay/28a7a336d0c05c53006e4b8a5154721392c589a3fe4523acc7b37f5e09e2747f/merged major:0 minor:440 fsType:overlay blockSize:0} overlay_0-442:{mountpoint:/var/lib/containers/storage/overlay/4e902d44dc498e0ee7084eb109f2066c3ba4271f7953b52e8686a6eaac16f1f9/merged major:0 minor:442 fsType:overlay blockSize:0} overlay_0-444:{mountpoint:/var/lib/containers/storage/overlay/baa0f2c8ee3139a448d70d7a76480ff8e08d626b3573664c90c77450dd69777c/merged major:0 minor:444 fsType:overlay blockSize:0} overlay_0-450:{mountpoint:/var/lib/containers/storage/overlay/284772a98a4e1b385afbe5ea525e578d732c7e38fb52f5b667141fc3894e928f/merged major:0 minor:450 fsType:overlay blockSize:0} overlay_0-455:{mountpoint:/var/lib/containers/storage/overlay/ccc9aea7f13cd21670f4e43843c286d3acc47b36130c3768b6c7d16d7bd4f164/merged major:0 minor:455 fsType:overlay blockSize:0} overlay_0-460:{mountpoint:/var/lib/containers/storage/overlay/dbfb013cd6c41d43fa1b72a57924857c76fc373c44fbcd5dc72724d1d4203073/merged major:0 minor:460 fsType:overlay blockSize:0} overlay_0-465:{mountpoint:/var/lib/containers/storage/overlay/e8ec62243404b299710a397d94e31b3a39701cdb598ee944bdc0e6bdbea051df/merged major:0 minor:465 fsType:overlay blockSize:0} overlay_0-482:{mountpoint:/var/lib/containers/storage/overlay/f3ff710d5cae5895a2674b1690f9c51d646dbf19bc4e93f8687c4ad81f708d69/merged major:0 minor:482 fsType:overlay blockSize:0} overlay_0-484:{mountpoint:/var/lib/containers/storage/overlay/9f140515864be2a1a65d9d4a8253f87829e7a31804c8eec7cc855172b5fbd149/merged major:0 minor:484 fsType:overlay blockSize:0} overlay_0-486:{mountpoint:/var/lib/containers/storage/overlay/2d8f81135cb5b04f4d70ad105392ebfe14ed8a6bcae643494fd7f4a3d287a1ed/merged major:0 minor:486 fsType:overlay blockSize:0} overlay_0-488:{mountpoint:/var/lib/containers/storage/overlay/735e5e3a31b33dea0ddf0d354fca2ffcf28a4fc12fdf1a9f573e3fe1ef6cd090/merged major:0 minor:488 fsType:overlay blockSize:0} overlay_0-490:{mountpoint:/var/lib/containers/storage/overlay/1e055f3a3f93f5e8d1d0fc1330c3a47ef54fc2ccd0ffbc4f473f8ec10bea7d70/merged major:0 minor:490 fsType:overlay blockSize:0} overlay_0-496:{mountpoint:/var/lib/containers/storage/overlay/dd15d3b968b0e65a787b9dc78508a24df7128ae2511ddc1687593b207855adf5/merged major:0 minor:496 fsType:overlay blockSize:0} overlay_0-501:{mountpoint:/var/lib/containers/storage/overlay/989421273be420a1728598e3280cd83ec54d85bce8a95d7e47927af8f3f241d5/merged major:0 minor:501 fsType:overlay blockSize:0} overlay_0-62:{mountpoint:/var/lib/containers/storage/overlay/724d43cfac538b5362f3f86528f21b6ef4e5edccfc59013eeb6f9ebf2aefb470/merged major:0 minor:62 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/e97e88948e964eb510bb26e445cf61c3537e22139075cfbc77630a447c2c92ed/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/931813bc9d309b1683284aa77bf9f6042015a2e97ea93921accc56ab0499ad22/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/2c02eb4367be65124c475bd0590053fb988968d3667836397df34b11bc2e175e/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/c8a3dae0145a3396241a66004be4d5047cd50bb0b381ebd2a427606f22b8b970/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/c0480a7615f4a52d7adc7bad7f82b5eab76eec86f6d95b5e31904dd08b6957f3/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/d5bfb44ad68b0e1a1d89daa698a400364125a7405953657a240dad3d57ed3c9f/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/7758e79c4a7232bc2fbd973af196034076029477eef79893d6e549a75c93f2b6/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/3f7f0a8f35bb4f3b68dc1834ccefbda61583cb996870f4932142c93c6c7c2da5/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/00acadf2072ef5262b23229ba98ea3884c16ade0de772f628478620a0256a697/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 30 18:33:52.535076 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.534566 6960 manager.go:217] Machine: {Timestamp:2026-10-30 18:33:52.534100464 +0000 UTC m=+0.063131612 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098586 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec26ec8f627b0e8f257c8877c68b21eb SystemUUID:ec26ec8f-627b-0e8f-257c-8877c68b21eb BootID:3b212ecd-8fbd-45c5-8b6e-b3b18acb359d Filesystems:[{Device:overlay_0-460 DeviceMajor:0 DeviceMinor:460 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b0f929fc-a9ea-4935-9fe7-770c5e7d4de5/volumes/kubernetes.io~projected/kube-api-access-4rncj DeviceMajor:0 DeviceMinor:56 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ebb7a6cb2f181cb9daf6993af1f5e8a539dc87cc81cd568fb2eb857afff8935c/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/158becf0-d6cf-4917-8329-4f64ae5a4ca4/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:331 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8f76499e-97ad-4c03-8a72-a62f65619596/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:424 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6c1405e1-dd94-4151-9b4f-6392155a2023/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:197 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2c0ae33ab021d9f5442b117896d2dab11ba19c65301da78cb6bad9ac5d0117dc/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/312dae04-dfda-41da-90ca-36b3b44a12ea/volumes/kubernetes.io~projected/kube-api-access-z8422 DeviceMajor:0 DeviceMinor:257 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/158becf0-d6cf-4917-8329-4f64ae5a4ca4/volumes/kubernetes.io~projected/kube-api-access-4zndl DeviceMajor:0 DeviceMinor:333 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-400 DeviceMajor:0 DeviceMinor:400 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-488 DeviceMajor:0 DeviceMinor:488 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:289 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b4961ad6-2fc3-4c01-a565-6f66068d2958/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:47 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-271 DeviceMajor:0 DeviceMinor:271 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-366 DeviceMajor:0 DeviceMinor:366 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fb66d280-cf24-4549-8fdb-89f16b6080c1/volumes/kubernetes.io~projected/kube-api-access-ztd9j DeviceMajor:0 DeviceMinor:477 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/90488e60-cb70-4afa-a54b-6f333d345e84/volumes/kubernetes.io~projected/kube-api-access-xzfgl DeviceMajor:0 DeviceMinor:52 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2069a85a-44e7-4c98-9f47-7fad4416c15b/volumes/kubernetes.io~projected/kube-api-access-wq2pp DeviceMajor:0 DeviceMinor:341 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-392 DeviceMajor:0 DeviceMinor:392 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/32522431fb329153a6eeebccc7680bd24dcdea651248be4cc3d5f50b8bb5a1e3/userdata/shm DeviceMajor:0 DeviceMinor:434 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-444 DeviceMajor:0 DeviceMinor:444 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0a1badb7-be53-4b7f-933f-23e2599ab5ba/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:45 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-216 DeviceMajor:0 DeviceMinor:216 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-218 DeviceMajor:0 DeviceMinor:218 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-277 DeviceMajor:0 DeviceMinor:277 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:287 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-342 DeviceMajor:0 DeviceMinor:342 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/56cf64efd808a743e34aa2da5f607cdce99f15da9da3be52befbd6612366c7d4/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d/volumes/kubernetes.io~projected/kube-api-access-25trq DeviceMajor:0 DeviceMinor:291 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-379 DeviceMajor:0 DeviceMinor:379 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8f76499e-97ad-4c03-8a72-a62f65619596/volumes/kubernetes.io~projected/kube-api-access-jrzxm DeviceMajor:0 DeviceMinor:416 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-265 DeviceMajor:0 DeviceMinor:265 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4f8841c0-4cf3-45d9-9301-957d4fffe1ce/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:252 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:279 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a2a9935a2a2d2b39234f40a39264586ed51525af2ed3b73ad1b2f5be53e581a7/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/81f52134-6a09-431f-9f80-412ae7bbead8/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/0a1badb7-be53-4b7f-933f-23e2599ab5ba/volumes/kubernetes.io~projected/kube-api-access-5pf82 DeviceMajor:0 DeviceMinor:203 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/90488e60-cb70-4afa-a54b-6f333d345e84/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:474 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-350 DeviceMajor:0 DeviceMinor:350 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/81f52134-6a09-431f-9f80-412ae7bbead8/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:48 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d2baf406-5501-4717-b5e1-d145edfe9ddc/volumes/kubernetes.io~projected/kube-api-access-m4s85 DeviceMajor:0 DeviceMinor:51 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8ae7cf2fa0bd11e84801e93bc28861efbd30ad203a94848ef0f5199e314f4f4c/userdata/shm DeviceMajor:0 DeviceMinor:70 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/158becf0-d6cf-4917-8329-4f64ae5a4ca4/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:330 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/683740f7ded55252e4418cbf3eef7b202800bc07ad67aee5e15472576a901d63/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/119b97a32ab1fa4705dfa33313f5a74d8b7b63668d39e87f373a39bbe1dd070e/userdata/shm DeviceMajor:0 DeviceMinor:334 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-386 DeviceMajor:0 DeviceMinor:386 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-403 DeviceMajor:0 DeviceMinor:403 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8f76499e-97ad-4c03-8a72-a62f65619596/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:428 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:284 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f7b465a0d6ebcaf211c2d96f236da176e968470550f58af269fdd08aa306e298/userdata/shm DeviceMajor:0 DeviceMinor:359 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:279 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8f76499e-97ad-4c03-8a72-a62f65619596/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:431 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8f76499e-97ad-4c03-8a72-a62f65619596/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:423 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/433ed181-b45e-45c0-a593-9cb0b6c09348/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:249 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/312dae04-dfda-41da-90ca-36b3b44a12ea/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:250 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-465 DeviceMajor:0 DeviceMinor:465 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-490 DeviceMajor:0 DeviceMinor:490 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a3060082255cf6b51651149218af4bdbba6f1b0033acf76294ab9f552e69ec40/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/4f8841c0-4cf3-45d9-9301-957d4fffe1ce/volumes/kubernetes.io~projected/kube-api-access-x9nwr DeviceMajor:0 DeviceMinor:258 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/158becf0-d6cf-4917-8329-4f64ae5a4ca4/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:328 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:279 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8f76499e-97ad-4c03-8a72-a62f65619596/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:425 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-482 DeviceMajor:0 DeviceMinor:482 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-501 DeviceMajor:0 DeviceMinor:501 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c9ae698e2ddbafabe23051c119ab3b29049da90efcd4e436acdec3095cd3fa73/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6c1405e1-dd94-4151-9b4f-6392155a2023/volumes/kubernetes.io~projected/kube-api-access-6z8lm DeviceMajor:0 DeviceMinor:202 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-269 DeviceMajor:0 DeviceMinor:269 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-397 DeviceMajor:0 DeviceMinor:397 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8f76499e-97ad-4c03-8a72-a62f65619596/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:423 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/91716cdd-99fb-4140-a52c-6d357ac74fb1/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:201 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/433ed181-b45e-45c0-a593-9cb0b6c09348/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:247 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/25377c15-6e29-46b3-9f1b-9b1fc38570f5/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:355 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8f76499e-97ad-4c03-8a72-a62f65619596/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:426 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/05cad8b6-56f5-4bd1-b12d-b2ae456b7027/volumes/kubernetes.io~projected/kube-api-access-vj8gn DeviceMajor:0 DeviceMinor:54 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/908390dea71dfceecda52d389bc886f06dadc8f7acb975b9402fe99fc931d605/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f653dd2bb9642410489171060113e275c78337a3f25db513e81c1722e9966388/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cf60a7a7a606fdc05b761a1d4e2408b15e4ec237e5a5c22d1b6a49bde2a24ad5/userdata/shm DeviceMajor:0 DeviceMinor:263 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:286 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-374 DeviceMajor:0 DeviceMinor:374 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b40da5c1f34f4f94eb250f3bed5492801d825f9ce4f5fe472b32b5fae787bd59/userdata/shm DeviceMajor:0 DeviceMinor:475 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-267 DeviceMajor:0 DeviceMinor:267 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/50b0bb5c3b75d6ed563d7acd7bc142c8e6e66f899b26bcdb63b7e097dea2f1fa/userdata/shm DeviceMajor:0 DeviceMinor:304 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-309 DeviceMajor:0 DeviceMinor:309 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-317 DeviceMajor:0 DeviceMinor:317 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-140 DeviceMajor:0 DeviceMinor:140 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/158becf0-d6cf-4917-8329-4f64ae5a4ca4/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:327 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e163025a-b19f-44e2-9058-b65f31bfbcc4/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:470 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-484 DeviceMajor:0 DeviceMinor:484 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/eef45986-9475-4e3e-a3f9-fdb682474ff3/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:49 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/4f8841c0-4cf3-45d9-9301-957d4fffe1ce/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:248 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/433ed181-b45e-45c0-a593-9cb0b6c09348/volumes/kubernetes.io~projected/kube-api-access-g52xm DeviceMajor:0 DeviceMinor:259 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-336 DeviceMajor:0 DeviceMinor:336 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2069a85a-44e7-4c98-9f47-7fad4416c15b/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:338 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8f76499e-97ad-4c03-8a72-a62f65619596/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:427 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-486 DeviceMajor:0 DeviceMinor:486 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/eef45986-9475-4e3e-a3f9-fdb682474ff3/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:50 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5b826abe-3f79-43ab-8813-2406d5c4dae9/volumes/kubernetes.io~projected/kube-api-access-kxzzq DeviceMajor:0 DeviceMinor:55 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2df7bb0c79d3346dc1213c978fbd1bf6cf2feee082618ee2378c05df610217e3/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:279 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:290 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-322 DeviceMajor:0 DeviceMinor:322 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/158becf0-d6cf-4917-8329-4f64ae5a4ca4/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:332 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/37d5c880b26121a7ab8d7ad2034e9521e55fd3ef0e30c419741bfabac5b7882a/userdata/shm DeviceMajor:0 DeviceMinor:364 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ef86d9fa-a344-4886-b80c-89ce84f418ea/volumes/kubernetes.io~projected/kube-api-access-l927s DeviceMajor:0 DeviceMinor:57 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-432 DeviceMajor:0 DeviceMinor:432 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8f76499e-97ad-4c03-8a72-a62f65619596/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:423 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-450 DeviceMajor:0 DeviceMinor:450 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/06687992306a30ad3c6f9044932282015ccabc988888a1c185e89fa381425904/userdata/shm DeviceMajor:0 DeviceMinor:480 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/e91ad8e0-05ce-420f-9594-445f4d48739c/volumes/kubernetes.io~projected/kube-api-access-nb2bf DeviceMajor:0 DeviceMinor:58 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3fc6a6002c134d13eff65b44fca6ed477f75262fb065531496148ca5cc8559e7/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8f76499e-97ad-4c03-8a72-a62f65619596/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:421 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8f76499e-97ad-4c03-8a72-a62f65619596/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:422 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-62 DeviceMajor:0 DeviceMinor:62 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/172feea2873431f0fbd887a83aadcc45053c61f7631d004fbfbf5d3f0aa4dd77/userdata/shm DeviceMajor:0 DeviceMinor:64 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/91716cdd-99fb-4140-a52c-6d357ac74fb1/volumes/kubernetes.io~projected/kube-api-access-bbvzt DeviceMajor:0 DeviceMinor:204 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:288 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/158becf0-d6cf-4917-8329-4f64ae5a4ca4/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:329 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2069a85a-44e7-4c98-9f47-7fad4416c15b/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:340 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-362 DeviceMajor:0 DeviceMinor:362 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/81f52134-6a09-431f-9f80-412ae7bbead8/volumes/kubernetes.io~projected/kube-api-access-rdz5l DeviceMajor:0 DeviceMinor:59 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-381 DeviceMajor:0 DeviceMinor:381 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8f76499e-97ad-4c03-8a72-a62f65619596/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:430 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-440 DeviceMajor:0 DeviceMinor:440 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-442 DeviceMajor:0 DeviceMinor:442 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/eef45986-9475-4e3e-a3f9-fdb682474ff3/volumes/kubernetes.io~projected/kube-api-access-knxck DeviceMajor:0 DeviceMinor:53 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/095a58f8a000a16f80609d89f8adf94a3432280b70698e057daf05d423517c6f/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/312dae04-dfda-41da-90ca-36b3b44a12ea/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:251 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/25377c15-6e29-46b3-9f1b-9b1fc38570f5/volumes/kubernetes.io~projected/kube-api-access-6546m DeviceMajor:0 DeviceMinor:361 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-411 DeviceMajor:0 DeviceMinor:411 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-455 DeviceMajor:0 DeviceMinor:455 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2069a85a-44e7-4c98-9f47-7fad4416c15b/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:339 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:285 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f4c144adff68eb30cf1418f5b0c943b73d737798daeb89a49a0e73db0db3b1db/userdata/shm DeviceMajor:0 DeviceMinor:478 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-496 DeviceMajor:0 DeviceMinor:496 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-111 DeviceMajor:0 DeviceMinor:111 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6f84cca17983b8b70b45b2bcdc32e64617a0f4bb6889b0c8fa47567b6a7149eb/userdata/shm DeviceMajor:0 DeviceMinor:261 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:283 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-311 DeviceMajor:0 DeviceMinor:311 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8f76499e-97ad-4c03-8a72-a62f65619596/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:429 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8f76499e-97ad-4c03-8a72-a62f65619596/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:423 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:285 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:06687992306a30a MacAddress:8a:c9:ac:e9:34:93 Speed:10000 Mtu:8901} {Name:095a58f8a000a16 MacAddress:8e:3e:6c:b2:28:a9 Speed:10000 Mtu:8901} {Name:119b97a32ab1fa4 MacAddress:5a:98:f6:0c:aa:68 Speed:10000 Mtu:8901} {Name:2c0ae33ab021d9f MacAddress:aa:c7:bd:ba:69:e7 Speed:10000 Mtu:8901} {Name:32522431fb32915 MacAddress:16:fd:1a:1d:b0:b1 Speed:10000 Mtu:8901} {Name:37d5c880b26121a MacAddress:4a:91:5a:bf:79:6f Speed:10000 Mtu:8901} {Name:50b0bb5c3b75d6e MacAddress:3e:f5:db:ec:d7:b6 Speed:10000 Mtu:8901} {Name:683740f7ded5525 MacAddress:2e:f8:b9:b7:2a:86 Speed:10000 Mtu:8901} {Name:6f84cca17983b8b MacAddress:9a:b4:07:b1:2d:35 Speed:10000 Mtu:8901} {Name:a3060082255cf6b MacAddress:8a:85:3a:e0:06:c1 Speed:10000 Mtu:8901} {Name:b40da5c1f34f4f9 MacAddress:3e:e1:09:3e:c2:34 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:df:c6:5f:8b Speed:0 Mtu:9001} {Name:br-int MacAddress:e6:fe:a1:2b:9a:ef Speed:0 Mtu:8901} {Name:ens5 MacAddress:02:ff:df:c6:5f:8b Speed:0 Mtu:9001} {Name:f4c144adff68eb3 MacAddress:26:22:3a:bc:ac:47 Speed:10000 Mtu:8901} {Name:f7b465a0d6ebcaf MacAddress:42:d0:09:6c:75:27 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:a2:e1:08:bd:94:6d Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:3e:ab:05:71:c4:54 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 18:33:52.535366 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.535080 6960 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 18:33:52.535366 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.535128 6960 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 18:33:52.535463 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.535444 6960 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 18:33:52.535592 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.535463 6960 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-163.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 18:33:52.535636 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.535594 6960 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 18:33:52.535636 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.535603 6960 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 18:33:52.535636 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.535612 6960 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 18:33:52.535636 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.535625 6960 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 18:33:52.535737 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.535710 6960 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 18:33:52.535864 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.535854 6960 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 18:33:52.535976 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.535965 6960 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 18:33:52.536017 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.536009 6960 kubelet.go:493] "Attempting to sync node with API server" Oct 30 18:33:52.536047 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.536024 6960 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 18:33:52.536047 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.536033 6960 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 18:33:52.536047 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.536041 6960 kubelet.go:405] "Adding apiserver pod source" Oct 30 18:33:52.536163 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.536049 6960 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 18:33:52.537266 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.537249 6960 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 18:33:52.537319 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.537267 6960 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 18:33:52.537871 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.537834 6960 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 18:33:52.540424 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.540409 6960 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 18:33:52.540424 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.540423 6960 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 18:33:52.540525 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.540507 6960 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 18:33:52.541071 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.541059 6960 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 18:33:52.541115 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.541089 6960 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 18:33:52.541272 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.541263 6960 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 18:33:52.541304 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.541278 6960 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 18:33:52.541304 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.541284 6960 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 18:33:52.541304 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.541293 6960 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 18:33:52.541304 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.541298 6960 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 18:33:52.541304 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.541304 6960 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 18:33:52.541441 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.541310 6960 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 18:33:52.541441 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.541315 6960 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 18:33:52.541441 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.541322 6960 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 18:33:52.541441 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.541328 6960 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 18:33:52.541441 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.541342 6960 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 18:33:52.541441 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.541351 6960 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 18:33:52.541441 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.541373 6960 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 18:33:52.541441 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.541379 6960 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 18:33:52.544842 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.544825 6960 server.go:1263] "Started kubelet" Oct 30 18:33:52.546751 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.544954 6960 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 18:33:52.546751 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.545058 6960 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 18:33:52.546751 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.545374 6960 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 18:33:52.546751 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.544913 6960 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 18:33:52.545404 ip-10-0-140-163 systemd[1]: Started Kubernetes Kubelet. Oct 30 18:33:52.547442 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.547424 6960 server.go:317] "Adding debug handlers to kubelet server" Oct 30 18:33:52.552327 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.552307 6960 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 18:33:52.552431 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.552372 6960 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 18:33:52.552719 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:33:52.552694 6960 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-163.us-east-2.compute.internal\" not found" Oct 30 18:33:52.552796 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.552742 6960 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 18:33:52.552796 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.552752 6960 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 18:33:52.552975 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.552960 6960 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 18:33:52.553109 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.552474 6960 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 18:27:44 +0000 UTC" deadline="2028-04-01 23:47:18.999966515 +0000 UTC" Oct 30 18:33:52.553238 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.553114 6960 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12461h13m26.44685787s" Oct 30 18:33:52.554551 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.554530 6960 factory.go:55] Registering systemd factory Oct 30 18:33:52.554551 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.554550 6960 factory.go:223] Registration of the systemd container factory successfully Oct 30 18:33:52.555571 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.555551 6960 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 18:33:52.556273 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.556235 6960 factory.go:153] Registering CRI-O factory Oct 30 18:33:52.556384 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.556372 6960 factory.go:223] Registration of the crio container factory successfully Oct 30 18:33:52.556709 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.556693 6960 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 18:33:52.556798 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.556721 6960 factory.go:103] Registering Raw factory Oct 30 18:33:52.556798 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.556739 6960 manager.go:1199] Started watching for new ooms in manager Oct 30 18:33:52.557019 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.556996 6960 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 18:33:52.557263 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.557244 6960 manager.go:319] Starting recovery of all containers Oct 30 18:33:52.558108 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.558090 6960 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 18:33:52.559189 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.556489 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2069a85a-44e7-4c98-9f47-7fad4416c15b" volumeName="kubernetes.io/configmap/2069a85a-44e7-4c98-9f47-7fad4416c15b-metrics-server-audit-profiles" seLinuxMountContext="" Oct 30 18:33:52.559283 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.559209 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="312dae04-dfda-41da-90ca-36b3b44a12ea" volumeName="kubernetes.io/secret/312dae04-dfda-41da-90ca-36b3b44a12ea-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 18:33:52.559283 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.559270 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c1405e1-dd94-4151-9b4f-6392155a2023" volumeName="kubernetes.io/secret/6c1405e1-dd94-4151-9b4f-6392155a2023-metrics-tls" seLinuxMountContext="" Oct 30 18:33:52.559384 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.559289 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f76499e-97ad-4c03-8a72-a62f65619596" volumeName="kubernetes.io/secret/8f76499e-97ad-4c03-8a72-a62f65619596-secret-grpc-tls" seLinuxMountContext="" Oct 30 18:33:52.559384 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.559304 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b0f929fc-a9ea-4935-9fe7-770c5e7d4de5" volumeName="kubernetes.io/configmap/b0f929fc-a9ea-4935-9fe7-770c5e7d4de5-serviceca" seLinuxMountContext="" Oct 30 18:33:52.559384 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.559334 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d2baf406-5501-4717-b5e1-d145edfe9ddc" volumeName="kubernetes.io/configmap/d2baf406-5501-4717-b5e1-d145edfe9ddc-multus-daemon-config" seLinuxMountContext="" Oct 30 18:33:52.559384 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.559350 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb66d280-cf24-4549-8fdb-89f16b6080c1" volumeName="kubernetes.io/projected/fb66d280-cf24-4549-8fdb-89f16b6080c1-kube-api-access-ztd9j" seLinuxMountContext="" Oct 30 18:33:52.559561 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.559466 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2069a85a-44e7-4c98-9f47-7fad4416c15b" volumeName="kubernetes.io/configmap/2069a85a-44e7-4c98-9f47-7fad4416c15b-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Oct 30 18:33:52.559561 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.559494 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="433ed181-b45e-45c0-a593-9cb0b6c09348" volumeName="kubernetes.io/secret/433ed181-b45e-45c0-a593-9cb0b6c09348-node-exporter-tls" seLinuxMountContext="" Oct 30 18:33:52.559561 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.559523 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4f8841c0-4cf3-45d9-9301-957d4fffe1ce" volumeName="kubernetes.io/secret/4f8841c0-4cf3-45d9-9301-957d4fffe1ce-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 18:33:52.559561 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.559542 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d" volumeName="kubernetes.io/secret/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext="" Oct 30 18:33:52.559561 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.559558 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81f52134-6a09-431f-9f80-412ae7bbead8" volumeName="kubernetes.io/configmap/81f52134-6a09-431f-9f80-412ae7bbead8-ovnkube-config" seLinuxMountContext="" Oct 30 18:33:52.559788 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.559579 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="158becf0-d6cf-4917-8329-4f64ae5a4ca4" volumeName="kubernetes.io/secret/158becf0-d6cf-4917-8329-4f64ae5a4ca4-secret-thanos-querier-tls" seLinuxMountContext="" Oct 30 18:33:52.559788 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.559598 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2069a85a-44e7-4c98-9f47-7fad4416c15b" volumeName="kubernetes.io/secret/2069a85a-44e7-4c98-9f47-7fad4416c15b-client-ca-bundle" seLinuxMountContext="" Oct 30 18:33:52.559788 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.559621 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4f8841c0-4cf3-45d9-9301-957d4fffe1ce" volumeName="kubernetes.io/empty-dir/4f8841c0-4cf3-45d9-9301-957d4fffe1ce-volume-directive-shadow" seLinuxMountContext="" Oct 30 18:33:52.559788 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.559636 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d" volumeName="kubernetes.io/projected/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-kube-api-access-25trq" seLinuxMountContext="" Oct 30 18:33:52.559788 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.559658 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f76499e-97ad-4c03-8a72-a62f65619596" volumeName="kubernetes.io/secret/8f76499e-97ad-4c03-8a72-a62f65619596-web-config" seLinuxMountContext="" Oct 30 18:33:52.559788 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.559682 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eef45986-9475-4e3e-a3f9-fdb682474ff3" volumeName="kubernetes.io/projected/eef45986-9475-4e3e-a3f9-fdb682474ff3-kube-api-access-knxck" seLinuxMountContext="" Oct 30 18:33:52.559788 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.559699 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef86d9fa-a344-4886-b80c-89ce84f418ea" volumeName="kubernetes.io/configmap/ef86d9fa-a344-4886-b80c-89ce84f418ea-iptables-alerter-script" seLinuxMountContext="" Oct 30 18:33:52.559788 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.559723 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="158becf0-d6cf-4917-8329-4f64ae5a4ca4" volumeName="kubernetes.io/secret/158becf0-d6cf-4917-8329-4f64ae5a4ca4-secret-grpc-tls" seLinuxMountContext="" Oct 30 18:33:52.559788 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.559741 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2069a85a-44e7-4c98-9f47-7fad4416c15b" volumeName="kubernetes.io/projected/2069a85a-44e7-4c98-9f47-7fad4416c15b-kube-api-access-wq2pp" seLinuxMountContext="" Oct 30 18:33:52.559788 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.559763 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="312dae04-dfda-41da-90ca-36b3b44a12ea" volumeName="kubernetes.io/projected/312dae04-dfda-41da-90ca-36b3b44a12ea-kube-api-access-z8422" seLinuxMountContext="" Oct 30 18:33:52.559788 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.559778 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4f8841c0-4cf3-45d9-9301-957d4fffe1ce" volumeName="kubernetes.io/configmap/4f8841c0-4cf3-45d9-9301-957d4fffe1ce-metrics-client-ca" seLinuxMountContext="" Oct 30 18:33:52.560212 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.559808 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4f8841c0-4cf3-45d9-9301-957d4fffe1ce" volumeName="kubernetes.io/projected/4f8841c0-4cf3-45d9-9301-957d4fffe1ce-kube-api-access-x9nwr" seLinuxMountContext="" Oct 30 18:33:52.560212 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.559830 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d" volumeName="kubernetes.io/secret/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 18:33:52.560212 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.559854 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d" volumeName="kubernetes.io/secret/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-cluster-tls-config" seLinuxMountContext="" Oct 30 18:33:52.560212 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.559872 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f76499e-97ad-4c03-8a72-a62f65619596" volumeName="kubernetes.io/configmap/8f76499e-97ad-4c03-8a72-a62f65619596-prometheus-k8s-rulefiles-0" seLinuxMountContext="" Oct 30 18:33:52.560212 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.559884 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="05cad8b6-56f5-4bd1-b12d-b2ae456b7027" volumeName="kubernetes.io/projected/05cad8b6-56f5-4bd1-b12d-b2ae456b7027-kube-api-access-vj8gn" seLinuxMountContext="" Oct 30 18:33:52.560212 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.559903 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a1badb7-be53-4b7f-933f-23e2599ab5ba" volumeName="kubernetes.io/projected/0a1badb7-be53-4b7f-933f-23e2599ab5ba-kube-api-access-5pf82" seLinuxMountContext="" Oct 30 18:33:52.561443 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.561415 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="158becf0-d6cf-4917-8329-4f64ae5a4ca4" volumeName="kubernetes.io/projected/158becf0-d6cf-4917-8329-4f64ae5a4ca4-kube-api-access-4zndl" seLinuxMountContext="" Oct 30 18:33:52.561542 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.561450 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="158becf0-d6cf-4917-8329-4f64ae5a4ca4" volumeName="kubernetes.io/secret/158becf0-d6cf-4917-8329-4f64ae5a4ca4-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 18:33:52.561542 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.561467 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="158becf0-d6cf-4917-8329-4f64ae5a4ca4" volumeName="kubernetes.io/secret/158becf0-d6cf-4917-8329-4f64ae5a4ca4-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext="" Oct 30 18:33:52.561542 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.561482 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25377c15-6e29-46b3-9f1b-9b1fc38570f5" volumeName="kubernetes.io/projected/25377c15-6e29-46b3-9f1b-9b1fc38570f5-kube-api-access-6546m" seLinuxMountContext="" Oct 30 18:33:52.561542 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.561517 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f76499e-97ad-4c03-8a72-a62f65619596" volumeName="kubernetes.io/configmap/8f76499e-97ad-4c03-8a72-a62f65619596-configmap-metrics-client-ca" seLinuxMountContext="" Oct 30 18:33:52.561542 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.561526 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f76499e-97ad-4c03-8a72-a62f65619596" volumeName="kubernetes.io/projected/8f76499e-97ad-4c03-8a72-a62f65619596-kube-api-access-jrzxm" seLinuxMountContext="" Oct 30 18:33:52.561801 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.561548 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b826abe-3f79-43ab-8813-2406d5c4dae9" volumeName="kubernetes.io/configmap/5b826abe-3f79-43ab-8813-2406d5c4dae9-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 18:33:52.561801 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.561555 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c1405e1-dd94-4151-9b4f-6392155a2023" volumeName="kubernetes.io/projected/6c1405e1-dd94-4151-9b4f-6392155a2023-kube-api-access-6z8lm" seLinuxMountContext="" Oct 30 18:33:52.561801 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.561563 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d" volumeName="kubernetes.io/secret/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext="" Oct 30 18:33:52.561801 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.561572 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f76499e-97ad-4c03-8a72-a62f65619596" volumeName="kubernetes.io/configmap/8f76499e-97ad-4c03-8a72-a62f65619596-prometheus-trusted-ca-bundle" seLinuxMountContext="" Oct 30 18:33:52.561801 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.561583 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f76499e-97ad-4c03-8a72-a62f65619596" volumeName="kubernetes.io/configmap/8f76499e-97ad-4c03-8a72-a62f65619596-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Oct 30 18:33:52.561801 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.561594 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f76499e-97ad-4c03-8a72-a62f65619596" volumeName="kubernetes.io/secret/8f76499e-97ad-4c03-8a72-a62f65619596-thanos-prometheus-http-client-file" seLinuxMountContext="" Oct 30 18:33:52.561801 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.561622 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="158becf0-d6cf-4917-8329-4f64ae5a4ca4" volumeName="kubernetes.io/secret/158becf0-d6cf-4917-8329-4f64ae5a4ca4-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext="" Oct 30 18:33:52.561801 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.561630 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="433ed181-b45e-45c0-a593-9cb0b6c09348" volumeName="kubernetes.io/configmap/433ed181-b45e-45c0-a593-9cb0b6c09348-metrics-client-ca" seLinuxMountContext="" Oct 30 18:33:52.561801 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.561637 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b826abe-3f79-43ab-8813-2406d5c4dae9" volumeName="kubernetes.io/configmap/5b826abe-3f79-43ab-8813-2406d5c4dae9-cni-binary-copy" seLinuxMountContext="" Oct 30 18:33:52.561801 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.561645 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d" volumeName="kubernetes.io/empty-dir/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-config-out" seLinuxMountContext="" Oct 30 18:33:52.561801 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.561652 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d" volumeName="kubernetes.io/secret/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-secret-alertmanager-main-tls" seLinuxMountContext="" Oct 30 18:33:52.561801 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.561664 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f76499e-97ad-4c03-8a72-a62f65619596" volumeName="kubernetes.io/secret/8f76499e-97ad-4c03-8a72-a62f65619596-config" seLinuxMountContext="" Oct 30 18:33:52.561801 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.561678 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="90488e60-cb70-4afa-a54b-6f333d345e84" volumeName="kubernetes.io/projected/90488e60-cb70-4afa-a54b-6f333d345e84-kube-api-access-xzfgl" seLinuxMountContext="" Oct 30 18:33:52.561801 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.561685 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="90488e60-cb70-4afa-a54b-6f333d345e84" volumeName="kubernetes.io/secret/90488e60-cb70-4afa-a54b-6f333d345e84-metrics-certs" seLinuxMountContext="" Oct 30 18:33:52.561801 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.561694 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c1405e1-dd94-4151-9b4f-6392155a2023" volumeName="kubernetes.io/configmap/6c1405e1-dd94-4151-9b4f-6392155a2023-config-volume" seLinuxMountContext="" Oct 30 18:33:52.561801 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.561702 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d" volumeName="kubernetes.io/projected/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-tls-assets" seLinuxMountContext="" Oct 30 18:33:52.561801 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.561709 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f76499e-97ad-4c03-8a72-a62f65619596" volumeName="kubernetes.io/configmap/8f76499e-97ad-4c03-8a72-a62f65619596-configmap-serving-certs-ca-bundle" seLinuxMountContext="" Oct 30 18:33:52.561801 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.561724 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f76499e-97ad-4c03-8a72-a62f65619596" volumeName="kubernetes.io/empty-dir/8f76499e-97ad-4c03-8a72-a62f65619596-config-out" seLinuxMountContext="" Oct 30 18:33:52.561801 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.561739 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e163025a-b19f-44e2-9058-b65f31bfbcc4" volumeName="kubernetes.io/secret/e163025a-b19f-44e2-9058-b65f31bfbcc4-original-pull-secret" seLinuxMountContext="" Oct 30 18:33:52.561801 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.561754 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e91ad8e0-05ce-420f-9594-445f4d48739c" volumeName="kubernetes.io/projected/e91ad8e0-05ce-420f-9594-445f4d48739c-kube-api-access-nb2bf" seLinuxMountContext="" Oct 30 18:33:52.561801 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.561761 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="158becf0-d6cf-4917-8329-4f64ae5a4ca4" volumeName="kubernetes.io/configmap/158becf0-d6cf-4917-8329-4f64ae5a4ca4-metrics-client-ca" seLinuxMountContext="" Oct 30 18:33:52.561801 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.561782 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="433ed181-b45e-45c0-a593-9cb0b6c09348" volumeName="kubernetes.io/empty-dir/433ed181-b45e-45c0-a593-9cb0b6c09348-node-exporter-textfile" seLinuxMountContext="" Oct 30 18:33:52.561801 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.561789 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="433ed181-b45e-45c0-a593-9cb0b6c09348" volumeName="kubernetes.io/projected/433ed181-b45e-45c0-a593-9cb0b6c09348-kube-api-access-g52xm" seLinuxMountContext="" Oct 30 18:33:52.561801 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.561797 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d" volumeName="kubernetes.io/secret/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-config-volume" seLinuxMountContext="" Oct 30 18:33:52.561801 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.561805 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81f52134-6a09-431f-9f80-412ae7bbead8" volumeName="kubernetes.io/configmap/81f52134-6a09-431f-9f80-412ae7bbead8-env-overrides" seLinuxMountContext="" Oct 30 18:33:52.561801 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.561813 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f76499e-97ad-4c03-8a72-a62f65619596" volumeName="kubernetes.io/empty-dir/8f76499e-97ad-4c03-8a72-a62f65619596-prometheus-k8s-db" seLinuxMountContext="" Oct 30 18:33:52.563214 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.561829 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f76499e-97ad-4c03-8a72-a62f65619596" volumeName="kubernetes.io/secret/8f76499e-97ad-4c03-8a72-a62f65619596-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext="" Oct 30 18:33:52.563214 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.561837 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f76499e-97ad-4c03-8a72-a62f65619596" volumeName="kubernetes.io/secret/8f76499e-97ad-4c03-8a72-a62f65619596-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 18:33:52.563214 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.561844 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a1badb7-be53-4b7f-933f-23e2599ab5ba" volumeName="kubernetes.io/empty-dir/0a1badb7-be53-4b7f-933f-23e2599ab5ba-data-volume" seLinuxMountContext="" Oct 30 18:33:52.563214 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.561851 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2069a85a-44e7-4c98-9f47-7fad4416c15b" volumeName="kubernetes.io/secret/2069a85a-44e7-4c98-9f47-7fad4416c15b-secret-metrics-server-client-certs" seLinuxMountContext="" Oct 30 18:33:52.563214 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.561859 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b826abe-3f79-43ab-8813-2406d5c4dae9" volumeName="kubernetes.io/configmap/5b826abe-3f79-43ab-8813-2406d5c4dae9-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 18:33:52.563214 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.561865 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d" volumeName="kubernetes.io/configmap/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-alertmanager-trusted-ca-bundle" seLinuxMountContext="" Oct 30 18:33:52.563214 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.561873 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91716cdd-99fb-4140-a52c-6d357ac74fb1" volumeName="kubernetes.io/projected/91716cdd-99fb-4140-a52c-6d357ac74fb1-kube-api-access-bbvzt" seLinuxMountContext="" Oct 30 18:33:52.563214 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.561880 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b4961ad6-2fc3-4c01-a565-6f66068d2958" volumeName="kubernetes.io/secret/b4961ad6-2fc3-4c01-a565-6f66068d2958-agent-certs" seLinuxMountContext="" Oct 30 18:33:52.563214 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.561887 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d2baf406-5501-4717-b5e1-d145edfe9ddc" volumeName="kubernetes.io/projected/d2baf406-5501-4717-b5e1-d145edfe9ddc-kube-api-access-m4s85" seLinuxMountContext="" Oct 30 18:33:52.563214 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.561898 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eef45986-9475-4e3e-a3f9-fdb682474ff3" volumeName="kubernetes.io/empty-dir/eef45986-9475-4e3e-a3f9-fdb682474ff3-etc-tuned" seLinuxMountContext="" Oct 30 18:33:52.563214 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.561911 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="05cad8b6-56f5-4bd1-b12d-b2ae456b7027" volumeName="kubernetes.io/empty-dir/05cad8b6-56f5-4bd1-b12d-b2ae456b7027-tmp-dir" seLinuxMountContext="" Oct 30 18:33:52.563214 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.561930 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a1badb7-be53-4b7f-933f-23e2599ab5ba" volumeName="kubernetes.io/configmap/0a1badb7-be53-4b7f-933f-23e2599ab5ba-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 30 18:33:52.563214 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.561946 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a1badb7-be53-4b7f-933f-23e2599ab5ba" volumeName="kubernetes.io/secret/0a1badb7-be53-4b7f-933f-23e2599ab5ba-insights-runtime-extractor-tls" seLinuxMountContext="" Oct 30 18:33:52.563214 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.561958 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4f8841c0-4cf3-45d9-9301-957d4fffe1ce" volumeName="kubernetes.io/secret/4f8841c0-4cf3-45d9-9301-957d4fffe1ce-kube-state-metrics-tls" seLinuxMountContext="" Oct 30 18:33:52.563214 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.561969 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d" volumeName="kubernetes.io/empty-dir/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-alertmanager-main-db" seLinuxMountContext="" Oct 30 18:33:52.563214 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.561981 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81f52134-6a09-431f-9f80-412ae7bbead8" volumeName="kubernetes.io/secret/81f52134-6a09-431f-9f80-412ae7bbead8-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 18:33:52.563214 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.561993 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f76499e-97ad-4c03-8a72-a62f65619596" volumeName="kubernetes.io/configmap/8f76499e-97ad-4c03-8a72-a62f65619596-prometheus-k8s-rulefiles-2" seLinuxMountContext="" Oct 30 18:33:52.563214 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.562006 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91716cdd-99fb-4140-a52c-6d357ac74fb1" volumeName="kubernetes.io/secret/91716cdd-99fb-4140-a52c-6d357ac74fb1-cert" seLinuxMountContext="" Oct 30 18:33:52.563214 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.562017 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="158becf0-d6cf-4917-8329-4f64ae5a4ca4" volumeName="kubernetes.io/empty-dir/158becf0-d6cf-4917-8329-4f64ae5a4ca4-tmp" seLinuxMountContext="" Oct 30 18:33:52.563214 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.562029 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="433ed181-b45e-45c0-a593-9cb0b6c09348" volumeName="kubernetes.io/configmap/433ed181-b45e-45c0-a593-9cb0b6c09348-node-exporter-accelerators-collector-config" seLinuxMountContext="" Oct 30 18:33:52.563214 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.562041 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b826abe-3f79-43ab-8813-2406d5c4dae9" volumeName="kubernetes.io/projected/5b826abe-3f79-43ab-8813-2406d5c4dae9-kube-api-access-kxzzq" seLinuxMountContext="" Oct 30 18:33:52.563214 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.562052 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c1405e1-dd94-4151-9b4f-6392155a2023" volumeName="kubernetes.io/empty-dir/6c1405e1-dd94-4151-9b4f-6392155a2023-tmp-dir" seLinuxMountContext="" Oct 30 18:33:52.563214 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.562080 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d" volumeName="kubernetes.io/configmap/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-metrics-client-ca" seLinuxMountContext="" Oct 30 18:33:52.563214 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.562092 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f76499e-97ad-4c03-8a72-a62f65619596" volumeName="kubernetes.io/projected/8f76499e-97ad-4c03-8a72-a62f65619596-tls-assets" seLinuxMountContext="" Oct 30 18:33:52.563214 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.562104 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f76499e-97ad-4c03-8a72-a62f65619596" volumeName="kubernetes.io/secret/8f76499e-97ad-4c03-8a72-a62f65619596-secret-prometheus-k8s-tls" seLinuxMountContext="" Oct 30 18:33:52.563214 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.562115 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b0f929fc-a9ea-4935-9fe7-770c5e7d4de5" volumeName="kubernetes.io/projected/b0f929fc-a9ea-4935-9fe7-770c5e7d4de5-kube-api-access-4rncj" seLinuxMountContext="" Oct 30 18:33:52.563214 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.562127 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="158becf0-d6cf-4917-8329-4f64ae5a4ca4" volumeName="kubernetes.io/secret/158becf0-d6cf-4917-8329-4f64ae5a4ca4-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext="" Oct 30 18:33:52.563214 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.562156 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="433ed181-b45e-45c0-a593-9cb0b6c09348" volumeName="kubernetes.io/secret/433ed181-b45e-45c0-a593-9cb0b6c09348-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 18:33:52.563214 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.562171 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81f52134-6a09-431f-9f80-412ae7bbead8" volumeName="kubernetes.io/projected/81f52134-6a09-431f-9f80-412ae7bbead8-kube-api-access-rdz5l" seLinuxMountContext="" Oct 30 18:33:52.563214 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.562191 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eef45986-9475-4e3e-a3f9-fdb682474ff3" volumeName="kubernetes.io/empty-dir/eef45986-9475-4e3e-a3f9-fdb682474ff3-tmp" seLinuxMountContext="" Oct 30 18:33:52.563214 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.562260 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25377c15-6e29-46b3-9f1b-9b1fc38570f5" volumeName="kubernetes.io/secret/25377c15-6e29-46b3-9f1b-9b1fc38570f5-monitoring-plugin-cert" seLinuxMountContext="" Oct 30 18:33:52.563214 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.562383 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="312dae04-dfda-41da-90ca-36b3b44a12ea" volumeName="kubernetes.io/secret/312dae04-dfda-41da-90ca-36b3b44a12ea-openshift-state-metrics-tls" seLinuxMountContext="" Oct 30 18:33:52.563214 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.562396 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4f8841c0-4cf3-45d9-9301-957d4fffe1ce" volumeName="kubernetes.io/configmap/4f8841c0-4cf3-45d9-9301-957d4fffe1ce-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext="" Oct 30 18:33:52.563214 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.562407 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d" volumeName="kubernetes.io/secret/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-web-config" seLinuxMountContext="" Oct 30 18:33:52.563214 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.562419 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81f52134-6a09-431f-9f80-412ae7bbead8" volumeName="kubernetes.io/configmap/81f52134-6a09-431f-9f80-412ae7bbead8-ovnkube-script-lib" seLinuxMountContext="" Oct 30 18:33:52.563214 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.562430 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f76499e-97ad-4c03-8a72-a62f65619596" volumeName="kubernetes.io/configmap/8f76499e-97ad-4c03-8a72-a62f65619596-prometheus-k8s-rulefiles-1" seLinuxMountContext="" Oct 30 18:33:52.563214 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.562441 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f76499e-97ad-4c03-8a72-a62f65619596" volumeName="kubernetes.io/secret/8f76499e-97ad-4c03-8a72-a62f65619596-secret-kube-rbac-proxy" seLinuxMountContext="" Oct 30 18:33:52.563214 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.562453 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d2baf406-5501-4717-b5e1-d145edfe9ddc" volumeName="kubernetes.io/configmap/d2baf406-5501-4717-b5e1-d145edfe9ddc-cni-binary-copy" seLinuxMountContext="" Oct 30 18:33:52.563214 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.562467 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2069a85a-44e7-4c98-9f47-7fad4416c15b" volumeName="kubernetes.io/secret/2069a85a-44e7-4c98-9f47-7fad4416c15b-secret-metrics-server-tls" seLinuxMountContext="" Oct 30 18:33:52.563214 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.562481 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="312dae04-dfda-41da-90ca-36b3b44a12ea" volumeName="kubernetes.io/configmap/312dae04-dfda-41da-90ca-36b3b44a12ea-metrics-client-ca" seLinuxMountContext="" Oct 30 18:33:52.563214 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.562521 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f76499e-97ad-4c03-8a72-a62f65619596" volumeName="kubernetes.io/secret/8f76499e-97ad-4c03-8a72-a62f65619596-secret-metrics-client-certs" seLinuxMountContext="" Oct 30 18:33:52.563214 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.562534 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef86d9fa-a344-4886-b80c-89ce84f418ea" volumeName="kubernetes.io/projected/ef86d9fa-a344-4886-b80c-89ce84f418ea-kube-api-access-l927s" seLinuxMountContext="" Oct 30 18:33:52.563214 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.562547 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2069a85a-44e7-4c98-9f47-7fad4416c15b" volumeName="kubernetes.io/empty-dir/2069a85a-44e7-4c98-9f47-7fad4416c15b-audit-log" seLinuxMountContext="" Oct 30 18:33:52.563214 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.562561 6960 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b4961ad6-2fc3-4c01-a565-6f66068d2958" volumeName="kubernetes.io/configmap/b4961ad6-2fc3-4c01-a565-6f66068d2958-konnectivity-ca" seLinuxMountContext="" Oct 30 18:33:52.563214 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.562573 6960 reconstruct.go:97] "Volume reconstruction finished" Oct 30 18:33:52.563214 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.562582 6960 reconciler.go:29] "Reconciler: start to sync state" Oct 30 18:33:52.566045 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.566026 6960 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 18:33:52.583341 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.583293 6960 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 18:33:52.584794 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.584652 6960 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 18:33:52.584794 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.584673 6960 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 18:33:52.584794 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.584691 6960 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 18:33:52.584794 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:33:52.584733 6960 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 18:33:52.589018 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.588996 6960 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 18:33:52.597092 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.597068 6960 generic.go:356] "Generic (PLEG): container finished" podID="7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d" containerID="e01e0e8685fc3c23441371b7c14e6a97b36e570510e3628e07b8c3e5f4bf2e84" exitCode=0 Oct 30 18:33:52.606181 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.606135 6960 generic.go:356] "Generic (PLEG): container finished" podID="5b826abe-3f79-43ab-8813-2406d5c4dae9" containerID="c246758d782ae07e82f94a156c6007334a401ef34106ea12fe38278878b79901" exitCode=0 Oct 30 18:33:52.606250 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.606184 6960 generic.go:356] "Generic (PLEG): container finished" podID="5b826abe-3f79-43ab-8813-2406d5c4dae9" containerID="e3060fff28d6443c7e4648c778be4126c6732ff40d8143db05f6a0db6b4ef4c3" exitCode=0 Oct 30 18:33:52.606250 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.606194 6960 generic.go:356] "Generic (PLEG): container finished" podID="5b826abe-3f79-43ab-8813-2406d5c4dae9" containerID="173a62de47c579d5b1ddd71171e22fb3ca597f987d7a14ecd9805e1200e2a7ed" exitCode=0 Oct 30 18:33:52.606250 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.606203 6960 generic.go:356] "Generic (PLEG): container finished" podID="5b826abe-3f79-43ab-8813-2406d5c4dae9" containerID="bf11d46cd8e7eaa012e2192c2590c0a17692a1a734704c7c1ba63b5fe9740e23" exitCode=0 Oct 30 18:33:52.606250 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.606211 6960 generic.go:356] "Generic (PLEG): container finished" podID="5b826abe-3f79-43ab-8813-2406d5c4dae9" containerID="7844f15920df2b73e16553ce0bae91c18981aed2246e8859c491973928bc3dc0" exitCode=0 Oct 30 18:33:52.606250 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.606218 6960 generic.go:356] "Generic (PLEG): container finished" podID="5b826abe-3f79-43ab-8813-2406d5c4dae9" containerID="800031197ad01291a6da93b42297c55fa1d72fe479eb4dec9a83ab5f7622ce35" exitCode=0 Oct 30 18:33:52.610887 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.610868 6960 generic.go:356] "Generic (PLEG): container finished" podID="40b5c76fae82f6d7a38a1a745d5d3947" containerID="da595342be50d698960624c3f93468dbb0cf59f511a95cb11bdf08cff27fc501" exitCode=0 Oct 30 18:33:52.619459 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.619436 6960 generic.go:356] "Generic (PLEG): container finished" podID="8f76499e-97ad-4c03-8a72-a62f65619596" containerID="696e8840b46d4d1acf76b61bafc7bc0d953821760feec22065f848f3daa8eaf7" exitCode=0 Oct 30 18:33:52.639101 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.639070 6960 generic.go:356] "Generic (PLEG): container finished" podID="433ed181-b45e-45c0-a593-9cb0b6c09348" containerID="00564a87b49da301f403b8a788c97f463906f216d56704076f1c53843f0fe6a8" exitCode=0 Oct 30 18:33:52.671488 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.671472 6960 manager.go:324] Recovery completed Oct 30 18:33:52.685012 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:33:52.684965 6960 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 30 18:33:52.693499 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.693484 6960 cpu_manager.go:226] "Starting" policy="none" Oct 30 18:33:52.693499 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.693497 6960 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 18:33:52.693607 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.693513 6960 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 18:33:52.693637 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.693616 6960 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 18:33:52.693637 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.693627 6960 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 18:33:52.693699 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.693640 6960 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 18:33:52.693699 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.693646 6960 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 18:33:52.693699 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.693651 6960 policy_none.go:50] "Start" Oct 30 18:33:52.693699 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.693660 6960 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 18:33:52.693699 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.693668 6960 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 18:33:52.693850 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.693779 6960 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 18:33:52.693850 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.693786 6960 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 18:33:52.693850 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.693791 6960 policy_none.go:44] "Start" Oct 30 18:33:52.700117 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.700085 6960 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 18:33:52.703279 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.703265 6960 manager.go:341] "Starting Device Plugin manager" Oct 30 18:33:52.703940 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:33:52.703288 6960 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 18:33:52.703982 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.703946 6960 server.go:87] "Starting device plugin registration server" Oct 30 18:33:52.704264 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.704252 6960 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 18:33:52.704297 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.704266 6960 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 18:33:52.704378 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.704360 6960 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 18:33:52.704517 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.704474 6960 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 18:33:52.704622 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.704522 6960 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 18:33:52.704622 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.704530 6960 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 18:33:52.704725 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.704580 6960 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T18:33:52.70451179Z","UUID":"9d6c4cd9-0efc-4b37-96a7-aa315f5114f5","Handler":null,"Name":"","Endpoint":""} Oct 30 18:33:52.705963 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.705947 6960 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 18:33:52.706048 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.705969 6960 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 18:33:52.706380 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:33:52.706357 6960 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 18:33:52.804837 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.804812 6960 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 18:33:52.806800 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.806784 6960 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-163.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 18:33:52.806877 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.806811 6960 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-163.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 18:33:52.806877 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.806822 6960 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-163.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 18:33:52.806877 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.806864 6960 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-140-163.us-east-2.compute.internal" Oct 30 18:33:52.818496 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.818476 6960 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-140-163.us-east-2.compute.internal" Oct 30 18:33:52.818570 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.818528 6960 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-140-163.us-east-2.compute.internal" Oct 30 18:33:52.885404 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.885366 6960 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-163.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-163.us-east-2.compute.internal"] Oct 30 18:33:52.885961 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.885933 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-163.us-east-2.compute.internal" event={"ID":"3b404058373133d538844156a64ac398","Type":"ContainerStarted","Data":"3a368fef34a48d4c7a8d51a5a8cd6471b7d622da94accf45cfad71e78230ef1b"} Oct 30 18:33:52.886011 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.885968 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-163.us-east-2.compute.internal" event={"ID":"3b404058373133d538844156a64ac398","Type":"ContainerStarted","Data":"172feea2873431f0fbd887a83aadcc45053c61f7631d004fbfbf5d3f0aa4dd77"} Oct 30 18:33:52.886011 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.885981 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-163.us-east-2.compute.internal" event={"ID":"40b5c76fae82f6d7a38a1a745d5d3947","Type":"ContainerStarted","Data":"603302b9d12c1d9fc525d9c3d19924af292b0e67809b0a7f2605a607abd8c559"} Oct 30 18:33:52.886011 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.885950 6960 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-163.us-east-2.compute.internal" Oct 30 18:33:52.886120 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.886085 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-163.us-east-2.compute.internal" event={"ID":"40b5c76fae82f6d7a38a1a745d5d3947","Type":"ContainerDied","Data":"da595342be50d698960624c3f93468dbb0cf59f511a95cb11bdf08cff27fc501"} Oct 30 18:33:52.886120 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.886108 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-163.us-east-2.compute.internal" event={"ID":"40b5c76fae82f6d7a38a1a745d5d3947","Type":"ContainerStarted","Data":"c9ae698e2ddbafabe23051c119ab3b29049da90efcd4e436acdec3095cd3fa73"} Oct 30 18:33:52.886212 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.886164 6960 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-163.us-east-2.compute.internal" Oct 30 18:33:52.900422 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.900407 6960 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 18:33:52.900476 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:33:52.900445 6960 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-140-163.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-163.us-east-2.compute.internal" Oct 30 18:33:52.900972 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.900956 6960 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 18:33:52.901040 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:33:52.901000 6960 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-140-163.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-163.us-east-2.compute.internal" Oct 30 18:33:52.968274 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.968206 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/40b5c76fae82f6d7a38a1a745d5d3947-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-163.us-east-2.compute.internal\" (UID: \"40b5c76fae82f6d7a38a1a745d5d3947\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-163.us-east-2.compute.internal" Oct 30 18:33:52.968274 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.968251 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/40b5c76fae82f6d7a38a1a745d5d3947-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-163.us-east-2.compute.internal\" (UID: \"40b5c76fae82f6d7a38a1a745d5d3947\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-163.us-east-2.compute.internal" Oct 30 18:33:52.968381 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:52.968281 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3b404058373133d538844156a64ac398-config\") pod \"kube-apiserver-proxy-ip-10-0-140-163.us-east-2.compute.internal\" (UID: \"3b404058373133d538844156a64ac398\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-163.us-east-2.compute.internal" Oct 30 18:33:53.068953 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.068930 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/40b5c76fae82f6d7a38a1a745d5d3947-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-163.us-east-2.compute.internal\" (UID: \"40b5c76fae82f6d7a38a1a745d5d3947\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-163.us-east-2.compute.internal" Oct 30 18:33:53.069038 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.068957 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/40b5c76fae82f6d7a38a1a745d5d3947-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-163.us-east-2.compute.internal\" (UID: \"40b5c76fae82f6d7a38a1a745d5d3947\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-163.us-east-2.compute.internal" Oct 30 18:33:53.069038 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.069003 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/40b5c76fae82f6d7a38a1a745d5d3947-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-163.us-east-2.compute.internal\" (UID: \"40b5c76fae82f6d7a38a1a745d5d3947\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-163.us-east-2.compute.internal" Oct 30 18:33:53.069038 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.069022 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3b404058373133d538844156a64ac398-config\") pod \"kube-apiserver-proxy-ip-10-0-140-163.us-east-2.compute.internal\" (UID: \"3b404058373133d538844156a64ac398\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-163.us-east-2.compute.internal" Oct 30 18:33:53.069207 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.069063 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3b404058373133d538844156a64ac398-config\") pod \"kube-apiserver-proxy-ip-10-0-140-163.us-east-2.compute.internal\" (UID: \"3b404058373133d538844156a64ac398\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-163.us-east-2.compute.internal" Oct 30 18:33:53.069207 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.069117 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/40b5c76fae82f6d7a38a1a745d5d3947-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-163.us-east-2.compute.internal\" (UID: \"40b5c76fae82f6d7a38a1a745d5d3947\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-163.us-east-2.compute.internal" Oct 30 18:33:53.536470 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.536436 6960 apiserver.go:52] "Watching apiserver" Oct 30 18:33:53.544252 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.544230 6960 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 18:33:53.544968 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.544947 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-r5d75","openshift-monitoring/thanos-querier-7c6f76d6d-k2h7h","openshift-multus/multus-68r7k","openshift-network-diagnostics/network-check-target-26f57","openshift-network-operator/iptables-alerter-b876b","openshift-monitoring/prometheus-k8s-0","kube-system/konnectivity-agent-4g8sn","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-163.us-east-2.compute.internal","openshift-insights/insights-runtime-extractor-w4clh","openshift-multus/network-metrics-daemon-dtppz","kube-system/global-pull-secret-syncer-zmdjg","openshift-monitoring/kube-state-metrics-5dc8d7d89b-7gk7g","openshift-ovn-kubernetes/ovnkube-node-czs4j","openshift-cluster-node-tuning-operator/tuned-rpzvk","openshift-dns/node-resolver-bk5br","openshift-image-registry/node-ca-qrdc6","openshift-multus/multus-additional-cni-plugins-nl4m8","openshift-monitoring/metrics-server-b6d7bcd58-z2xvj","openshift-ingress-canary/ingress-canary-bz2mv","openshift-monitoring/openshift-state-metrics-7bd7489957-8698b","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtz4g","openshift-monitoring/alertmanager-main-0","openshift-dns/dns-default-vfn4g","openshift-monitoring/node-exporter-hh862","kube-system/kube-apiserver-proxy-ip-10-0-140-163.us-east-2.compute.internal"] Oct 30 18:33:53.550462 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.550444 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 18:33:53.550684 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.550667 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 18:33:53.550863 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.550844 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 18:33:53.551529 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.551513 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 18:33:53.551956 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.551938 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 18:33:53.552083 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.552058 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-fgz4w\"" Oct 30 18:33:53.552198 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.552099 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-lc92h\"" Oct 30 18:33:53.552198 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.552180 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-9fr66\"" Oct 30 18:33:53.557270 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.557249 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-fshmx\"" Oct 30 18:33:53.557452 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.557424 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-qwdqr\"" Oct 30 18:33:53.557603 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.557587 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-9qttr\"" Oct 30 18:33:53.557716 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.557695 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-82xdm\"" Oct 30 18:33:53.557789 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.557741 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-jhsz4\"" Oct 30 18:33:53.557789 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.557741 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 18:33:53.557789 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.557669 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 18:33:53.557882 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.557801 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-bqrtm\"" Oct 30 18:33:53.557882 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.557830 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-pg8bd\"" Oct 30 18:33:53.558016 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.557999 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 18:33:53.558081 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.558067 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 18:33:53.558182 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.558115 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-s26z4\"" Oct 30 18:33:53.558425 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.558409 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 18:33:53.558487 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.558440 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-z87v4\"" Oct 30 18:33:53.558487 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.558482 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-g44rk\"" Oct 30 18:33:53.558570 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.558514 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-59qb5\"" Oct 30 18:33:53.558570 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.558554 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 18:33:53.558654 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.558601 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-6twlp\"" Oct 30 18:33:53.559711 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.559695 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 18:33:53.562225 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.562208 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 18:33:53.564508 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.564487 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 18:33:53.564628 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.564541 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 18:33:53.564628 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.564541 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 18:33:53.564744 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.564701 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 18:33:53.564744 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.564717 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 18:33:53.564948 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.564936 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 18:33:53.565508 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.565493 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 18:33:53.565548 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.565528 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 18:33:53.565717 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.565706 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 18:33:53.565997 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.565982 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 18:33:53.566066 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.566028 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 18:33:53.566066 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.566059 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 18:33:53.567417 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.567401 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 18:33:53.567458 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.567428 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 18:33:53.567634 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.567623 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 18:33:53.567678 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.567624 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 18:33:53.567963 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.567947 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 18:33:53.568211 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.568197 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 18:33:53.568289 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.568223 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 18:33:53.568289 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.568255 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 18:33:53.568289 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.568267 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 18:33:53.568289 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.568277 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 18:33:53.568289 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.568283 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 18:33:53.568540 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.568294 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 18:33:53.568540 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.568449 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 18:33:53.568982 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.568965 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 18:33:53.569084 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.569009 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 18:33:53.569084 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.569009 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 30 18:33:53.569459 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.569447 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 30 18:33:53.569946 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.569927 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 18:33:53.569946 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.569940 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 18:33:53.570070 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.569944 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 18:33:53.570070 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.569930 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 18:33:53.570070 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.569956 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 18:33:53.570070 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.569979 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 18:33:53.570070 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.569992 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 18:33:53.570263 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.570247 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 18:33:53.570313 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.570267 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d2baf406-5501-4717-b5e1-d145edfe9ddc-host-run-netns\") pod \"multus-68r7k\" (UID: \"d2baf406-5501-4717-b5e1-d145edfe9ddc\") " pod="openshift-multus/multus-68r7k" Oct 30 18:33:53.570313 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.570282 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 30 18:33:53.570313 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.570303 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/eef45986-9475-4e3e-a3f9-fdb682474ff3-etc-tuned\") pod \"tuned-rpzvk\" (UID: \"eef45986-9475-4e3e-a3f9-fdb682474ff3\") " pod="openshift-cluster-node-tuning-operator/tuned-rpzvk" Oct 30 18:33:53.570450 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.570330 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81f52134-6a09-431f-9f80-412ae7bbead8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:33:53.570450 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.570360 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d2baf406-5501-4717-b5e1-d145edfe9ddc-host-run-multus-certs\") pod \"multus-68r7k\" (UID: \"d2baf406-5501-4717-b5e1-d145edfe9ddc\") " pod="openshift-multus/multus-68r7k" Oct 30 18:33:53.570450 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.570394 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzfgl\" (UniqueName: \"kubernetes.io/projected/90488e60-cb70-4afa-a54b-6f333d345e84-kube-api-access-xzfgl\") pod \"network-metrics-daemon-dtppz\" (UID: \"90488e60-cb70-4afa-a54b-6f333d345e84\") " pod="openshift-multus/network-metrics-daemon-dtppz" Oct 30 18:33:53.570450 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.570423 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b0f929fc-a9ea-4935-9fe7-770c5e7d4de5-host\") pod \"node-ca-qrdc6\" (UID: \"b0f929fc-a9ea-4935-9fe7-770c5e7d4de5\") " pod="openshift-image-registry/node-ca-qrdc6" Oct 30 18:33:53.570450 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.570430 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/eef45986-9475-4e3e-a3f9-fdb682474ff3-etc-tuned\") pod \"tuned-rpzvk\" (UID: \"eef45986-9475-4e3e-a3f9-fdb682474ff3\") " pod="openshift-cluster-node-tuning-operator/tuned-rpzvk" Oct 30 18:33:53.570736 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.570473 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0a1badb7-be53-4b7f-933f-23e2599ab5ba-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w4clh\" (UID: \"0a1badb7-be53-4b7f-933f-23e2599ab5ba\") " pod="openshift-insights/insights-runtime-extractor-w4clh" Oct 30 18:33:53.570736 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.570500 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-rbnh2\"" Oct 30 18:33:53.570736 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.570522 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 30 18:33:53.570736 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.570553 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d2baf406-5501-4717-b5e1-d145edfe9ddc-multus-daemon-config\") pod \"multus-68r7k\" (UID: \"d2baf406-5501-4717-b5e1-d145edfe9ddc\") " pod="openshift-multus/multus-68r7k" Oct 30 18:33:53.570736 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.570589 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 18:33:53.570736 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.570612 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/eef45986-9475-4e3e-a3f9-fdb682474ff3-etc-sysctl-conf\") pod \"tuned-rpzvk\" (UID: \"eef45986-9475-4e3e-a3f9-fdb682474ff3\") " pod="openshift-cluster-node-tuning-operator/tuned-rpzvk" Oct 30 18:33:53.570736 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.570715 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0a1badb7-be53-4b7f-933f-23e2599ab5ba-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-w4clh\" (UID: \"0a1badb7-be53-4b7f-933f-23e2599ab5ba\") " pod="openshift-insights/insights-runtime-extractor-w4clh" Oct 30 18:33:53.571069 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.570726 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b0f929fc-a9ea-4935-9fe7-770c5e7d4de5-serviceca\") pod \"node-ca-qrdc6\" (UID: \"b0f929fc-a9ea-4935-9fe7-770c5e7d4de5\") " pod="openshift-image-registry/node-ca-qrdc6" Oct 30 18:33:53.571069 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.570767 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 18:33:53.571069 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.570782 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d2baf406-5501-4717-b5e1-d145edfe9ddc-multus-daemon-config\") pod \"multus-68r7k\" (UID: \"d2baf406-5501-4717-b5e1-d145edfe9ddc\") " pod="openshift-multus/multus-68r7k" Oct 30 18:33:53.571069 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.570795 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 18:33:53.571069 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.570836 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b4961ad6-2fc3-4c01-a565-6f66068d2958-konnectivity-ca\") pod \"konnectivity-agent-4g8sn\" (UID: \"b4961ad6-2fc3-4c01-a565-6f66068d2958\") " pod="kube-system/konnectivity-agent-4g8sn" Oct 30 18:33:53.571069 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.570920 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/81f52134-6a09-431f-9f80-412ae7bbead8-ovnkube-script-lib\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:33:53.571069 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.570958 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 30 18:33:53.571069 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.570980 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d2baf406-5501-4717-b5e1-d145edfe9ddc-etc-kubernetes\") pod \"multus-68r7k\" (UID: \"d2baf406-5501-4717-b5e1-d145edfe9ddc\") " pod="openshift-multus/multus-68r7k" Oct 30 18:33:53.571069 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.571002 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-vsdjx\"" Oct 30 18:33:53.571069 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.571006 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e91ad8e0-05ce-420f-9594-445f4d48739c-socket-dir\") pod \"aws-ebs-csi-driver-node-qtz4g\" (UID: \"e91ad8e0-05ce-420f-9594-445f4d48739c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtz4g" Oct 30 18:33:53.571069 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.571042 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-hh248\"" Oct 30 18:33:53.571069 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.571032 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e91ad8e0-05ce-420f-9594-445f4d48739c-registration-dir\") pod \"aws-ebs-csi-driver-node-qtz4g\" (UID: \"e91ad8e0-05ce-420f-9594-445f4d48739c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtz4g" Oct 30 18:33:53.571069 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.571053 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b4961ad6-2fc3-4c01-a565-6f66068d2958-konnectivity-ca\") pod \"konnectivity-agent-4g8sn\" (UID: \"b4961ad6-2fc3-4c01-a565-6f66068d2958\") " pod="kube-system/konnectivity-agent-4g8sn" Oct 30 18:33:53.571558 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.571177 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 18:33:53.571558 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.571085 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0a1badb7-be53-4b7f-933f-23e2599ab5ba-crio-socket\") pod \"insights-runtime-extractor-w4clh\" (UID: \"0a1badb7-be53-4b7f-933f-23e2599ab5ba\") " pod="openshift-insights/insights-runtime-extractor-w4clh" Oct 30 18:33:53.571558 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.571230 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4s85\" (UniqueName: \"kubernetes.io/projected/d2baf406-5501-4717-b5e1-d145edfe9ddc-kube-api-access-m4s85\") pod \"multus-68r7k\" (UID: \"d2baf406-5501-4717-b5e1-d145edfe9ddc\") " pod="openshift-multus/multus-68r7k" Oct 30 18:33:53.571558 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.571243 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/81f52134-6a09-431f-9f80-412ae7bbead8-ovnkube-script-lib\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:33:53.571558 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.571258 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5b826abe-3f79-43ab-8813-2406d5c4dae9-cnibin\") pod \"multus-additional-cni-plugins-nl4m8\" (UID: \"5b826abe-3f79-43ab-8813-2406d5c4dae9\") " pod="openshift-multus/multus-additional-cni-plugins-nl4m8" Oct 30 18:33:53.571558 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.571285 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5b826abe-3f79-43ab-8813-2406d5c4dae9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nl4m8\" (UID: \"5b826abe-3f79-43ab-8813-2406d5c4dae9\") " pod="openshift-multus/multus-additional-cni-plugins-nl4m8" Oct 30 18:33:53.571558 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.571314 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/eef45986-9475-4e3e-a3f9-fdb682474ff3-etc-sysctl-d\") pod \"tuned-rpzvk\" (UID: \"eef45986-9475-4e3e-a3f9-fdb682474ff3\") " pod="openshift-cluster-node-tuning-operator/tuned-rpzvk" Oct 30 18:33:53.571558 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.571358 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-7hx8m\"" Oct 30 18:33:53.571558 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.571391 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b0f929fc-a9ea-4935-9fe7-770c5e7d4de5-serviceca\") pod \"node-ca-qrdc6\" (UID: \"b0f929fc-a9ea-4935-9fe7-770c5e7d4de5\") " pod="openshift-image-registry/node-ca-qrdc6" Oct 30 18:33:53.571558 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.571403 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 30 18:33:53.571558 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.571355 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 18:33:53.571558 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.571338 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/eef45986-9475-4e3e-a3f9-fdb682474ff3-etc-systemd\") pod \"tuned-rpzvk\" (UID: \"eef45986-9475-4e3e-a3f9-fdb682474ff3\") " pod="openshift-cluster-node-tuning-operator/tuned-rpzvk" Oct 30 18:33:53.571558 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.571443 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-h4dl6\"" Oct 30 18:33:53.571558 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.571448 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eef45986-9475-4e3e-a3f9-fdb682474ff3-sys\") pod \"tuned-rpzvk\" (UID: \"eef45986-9475-4e3e-a3f9-fdb682474ff3\") " pod="openshift-cluster-node-tuning-operator/tuned-rpzvk" Oct 30 18:33:53.571558 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.571512 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e163025a-b19f-44e2-9058-b65f31bfbcc4-original-pull-secret\") pod \"global-pull-secret-syncer-zmdjg\" (UID: \"e163025a-b19f-44e2-9058-b65f31bfbcc4\") " pod="kube-system/global-pull-secret-syncer-zmdjg" Oct 30 18:33:53.571558 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.571540 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/81f52134-6a09-431f-9f80-412ae7bbead8-host-kubelet\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:33:53.571558 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.571544 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 30 18:33:53.571558 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.571549 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 30 18:33:53.571558 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.571563 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/81f52134-6a09-431f-9f80-412ae7bbead8-systemd-units\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:33:53.572395 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.571582 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5b826abe-3f79-43ab-8813-2406d5c4dae9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nl4m8\" (UID: \"5b826abe-3f79-43ab-8813-2406d5c4dae9\") " pod="openshift-multus/multus-additional-cni-plugins-nl4m8" Oct 30 18:33:53.572395 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.571600 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 30 18:33:53.572395 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.571604 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/eef45986-9475-4e3e-a3f9-fdb682474ff3-etc-modprobe-d\") pod \"tuned-rpzvk\" (UID: \"eef45986-9475-4e3e-a3f9-fdb682474ff3\") " pod="openshift-cluster-node-tuning-operator/tuned-rpzvk" Oct 30 18:33:53.572395 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.571612 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5b826abe-3f79-43ab-8813-2406d5c4dae9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nl4m8\" (UID: \"5b826abe-3f79-43ab-8813-2406d5c4dae9\") " pod="openshift-multus/multus-additional-cni-plugins-nl4m8" Oct 30 18:33:53.572395 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.571625 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eef45986-9475-4e3e-a3f9-fdb682474ff3-tmp\") pod \"tuned-rpzvk\" (UID: \"eef45986-9475-4e3e-a3f9-fdb682474ff3\") " pod="openshift-cluster-node-tuning-operator/tuned-rpzvk" Oct 30 18:33:53.572395 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.571660 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 18:33:53.572395 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.571670 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/81f52134-6a09-431f-9f80-412ae7bbead8-host-cni-bin\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:33:53.572395 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.571719 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eef45986-9475-4e3e-a3f9-fdb682474ff3-tmp\") pod \"tuned-rpzvk\" (UID: \"eef45986-9475-4e3e-a3f9-fdb682474ff3\") " pod="openshift-cluster-node-tuning-operator/tuned-rpzvk" Oct 30 18:33:53.572395 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.571750 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/81f52134-6a09-431f-9f80-412ae7bbead8-ovn-node-metrics-cert\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:33:53.572395 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.571768 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 30 18:33:53.572395 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.571787 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d2baf406-5501-4717-b5e1-d145edfe9ddc-multus-socket-dir-parent\") pod \"multus-68r7k\" (UID: \"d2baf406-5501-4717-b5e1-d145edfe9ddc\") " pod="openshift-multus/multus-68r7k" Oct 30 18:33:53.572395 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.571831 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 18:33:53.572395 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.571842 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 18:33:53.572395 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.571914 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d2baf406-5501-4717-b5e1-d145edfe9ddc-host-var-lib-kubelet\") pod \"multus-68r7k\" (UID: \"d2baf406-5501-4717-b5e1-d145edfe9ddc\") " pod="openshift-multus/multus-68r7k" Oct 30 18:33:53.572395 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.571938 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d2baf406-5501-4717-b5e1-d145edfe9ddc-hostroot\") pod \"multus-68r7k\" (UID: \"d2baf406-5501-4717-b5e1-d145edfe9ddc\") " pod="openshift-multus/multus-68r7k" Oct 30 18:33:53.572395 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.571961 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eef45986-9475-4e3e-a3f9-fdb682474ff3-lib-modules\") pod \"tuned-rpzvk\" (UID: \"eef45986-9475-4e3e-a3f9-fdb682474ff3\") " pod="openshift-cluster-node-tuning-operator/tuned-rpzvk" Oct 30 18:33:53.572395 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.571982 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 18:33:53.572395 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.571984 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eef45986-9475-4e3e-a3f9-fdb682474ff3-host\") pod \"tuned-rpzvk\" (UID: \"eef45986-9475-4e3e-a3f9-fdb682474ff3\") " pod="openshift-cluster-node-tuning-operator/tuned-rpzvk" Oct 30 18:33:53.572395 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.571994 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e163025a-b19f-44e2-9058-b65f31bfbcc4-original-pull-secret\") pod \"global-pull-secret-syncer-zmdjg\" (UID: \"e163025a-b19f-44e2-9058-b65f31bfbcc4\") " pod="kube-system/global-pull-secret-syncer-zmdjg" Oct 30 18:33:53.572395 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.572008 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ef86d9fa-a344-4886-b80c-89ce84f418ea-host-slash\") pod \"iptables-alerter-b876b\" (UID: \"ef86d9fa-a344-4886-b80c-89ce84f418ea\") " pod="openshift-network-operator/iptables-alerter-b876b" Oct 30 18:33:53.572395 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.572065 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05cad8b6-56f5-4bd1-b12d-b2ae456b7027-tmp-dir\") pod \"node-resolver-bk5br\" (UID: \"05cad8b6-56f5-4bd1-b12d-b2ae456b7027\") " pod="openshift-dns/node-resolver-bk5br" Oct 30 18:33:53.572395 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.572071 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/81f52134-6a09-431f-9f80-412ae7bbead8-ovn-node-metrics-cert\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:33:53.572395 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.572091 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/81f52134-6a09-431f-9f80-412ae7bbead8-log-socket\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:33:53.572395 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.572119 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d2baf406-5501-4717-b5e1-d145edfe9ddc-multus-conf-dir\") pod \"multus-68r7k\" (UID: \"d2baf406-5501-4717-b5e1-d145edfe9ddc\") " pod="openshift-multus/multus-68r7k" Oct 30 18:33:53.572395 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.572163 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z8lm\" (UniqueName: \"kubernetes.io/projected/6c1405e1-dd94-4151-9b4f-6392155a2023-kube-api-access-6z8lm\") pod \"dns-default-vfn4g\" (UID: \"6c1405e1-dd94-4151-9b4f-6392155a2023\") " pod="openshift-dns/dns-default-vfn4g" Oct 30 18:33:53.572395 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.572167 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05cad8b6-56f5-4bd1-b12d-b2ae456b7027-tmp-dir\") pod \"node-resolver-bk5br\" (UID: \"05cad8b6-56f5-4bd1-b12d-b2ae456b7027\") " pod="openshift-dns/node-resolver-bk5br" Oct 30 18:33:53.572395 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.572207 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/81f52134-6a09-431f-9f80-412ae7bbead8-var-lib-openvswitch\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:33:53.572395 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.572240 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8422\" (UniqueName: \"kubernetes.io/projected/312dae04-dfda-41da-90ca-36b3b44a12ea-kube-api-access-z8422\") pod \"openshift-state-metrics-7bd7489957-8698b\" (UID: \"312dae04-dfda-41da-90ca-36b3b44a12ea\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8698b" Oct 30 18:33:53.572395 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.572342 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 18:33:53.572395 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.572397 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 18:33:53.572395 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.572400 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb2bf\" (UniqueName: \"kubernetes.io/projected/e91ad8e0-05ce-420f-9594-445f4d48739c-kube-api-access-nb2bf\") pod \"aws-ebs-csi-driver-node-qtz4g\" (UID: \"e91ad8e0-05ce-420f-9594-445f4d48739c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtz4g" Oct 30 18:33:53.573718 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.572423 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-5vglk\"" Oct 30 18:33:53.573718 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.572443 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e163025a-b19f-44e2-9058-b65f31bfbcc4-dbus\") pod \"global-pull-secret-syncer-zmdjg\" (UID: \"e163025a-b19f-44e2-9058-b65f31bfbcc4\") " pod="kube-system/global-pull-secret-syncer-zmdjg" Oct 30 18:33:53.573718 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.572468 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/81f52134-6a09-431f-9f80-412ae7bbead8-run-ovn\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:33:53.573718 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.572472 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 18:33:53.573718 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.572493 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/81f52134-6a09-431f-9f80-412ae7bbead8-node-log\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:33:53.573718 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.572502 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 30 18:33:53.573718 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.572470 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 18:33:53.573718 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.572522 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 18:33:53.573718 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.572534 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/81f52134-6a09-431f-9f80-412ae7bbead8-env-overrides\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:33:53.573718 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.572553 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-4n8xm\"" Oct 30 18:33:53.573718 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.572571 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-8b4gujb9r0sdv\"" Oct 30 18:33:53.573718 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.572573 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d2baf406-5501-4717-b5e1-d145edfe9ddc-cni-binary-copy\") pod \"multus-68r7k\" (UID: \"d2baf406-5501-4717-b5e1-d145edfe9ddc\") " pod="openshift-multus/multus-68r7k" Oct 30 18:33:53.573718 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.572601 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 30 18:33:53.573718 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.572605 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-b062eu2aaotn4\"" Oct 30 18:33:53.573718 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.572477 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 30 18:33:53.573718 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.572619 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 18:33:53.573718 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.572640 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 30 18:33:53.573718 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.572598 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d2baf406-5501-4717-b5e1-d145edfe9ddc-host-var-lib-cni-multus\") pod \"multus-68r7k\" (UID: \"d2baf406-5501-4717-b5e1-d145edfe9ddc\") " pod="openshift-multus/multus-68r7k" Oct 30 18:33:53.573718 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.572663 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-xsppx\"" Oct 30 18:33:53.573718 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.572695 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 18:33:53.573718 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.572691 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eef45986-9475-4e3e-a3f9-fdb682474ff3-var-lib-kubelet\") pod \"tuned-rpzvk\" (UID: \"eef45986-9475-4e3e-a3f9-fdb682474ff3\") " pod="openshift-cluster-node-tuning-operator/tuned-rpzvk" Oct 30 18:33:53.573718 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.572718 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/81f52134-6a09-431f-9f80-412ae7bbead8-env-overrides\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:33:53.573718 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.572702 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 18:33:53.573718 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.572756 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e91ad8e0-05ce-420f-9594-445f4d48739c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qtz4g\" (UID: \"e91ad8e0-05ce-420f-9594-445f4d48739c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtz4g" Oct 30 18:33:53.573718 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.572799 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b4961ad6-2fc3-4c01-a565-6f66068d2958-agent-certs\") pod \"konnectivity-agent-4g8sn\" (UID: \"b4961ad6-2fc3-4c01-a565-6f66068d2958\") " pod="kube-system/konnectivity-agent-4g8sn" Oct 30 18:33:53.573718 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.572844 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 18:33:53.573718 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.572846 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6c1405e1-dd94-4151-9b4f-6392155a2023-metrics-tls\") pod \"dns-default-vfn4g\" (UID: \"6c1405e1-dd94-4151-9b4f-6392155a2023\") " pod="openshift-dns/dns-default-vfn4g" Oct 30 18:33:53.573718 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.572875 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/81f52134-6a09-431f-9f80-412ae7bbead8-host-run-netns\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:33:53.573718 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.572906 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d2baf406-5501-4717-b5e1-d145edfe9ddc-host-var-lib-cni-bin\") pod \"multus-68r7k\" (UID: \"d2baf406-5501-4717-b5e1-d145edfe9ddc\") " pod="openshift-multus/multus-68r7k" Oct 30 18:33:53.573718 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.572928 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90488e60-cb70-4afa-a54b-6f333d345e84-metrics-certs\") pod \"network-metrics-daemon-dtppz\" (UID: \"90488e60-cb70-4afa-a54b-6f333d345e84\") " pod="openshift-multus/network-metrics-daemon-dtppz" Oct 30 18:33:53.573718 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.572966 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/eef45986-9475-4e3e-a3f9-fdb682474ff3-etc-sysconfig\") pod \"tuned-rpzvk\" (UID: \"eef45986-9475-4e3e-a3f9-fdb682474ff3\") " pod="openshift-cluster-node-tuning-operator/tuned-rpzvk" Oct 30 18:33:53.573718 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.573188 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6c1405e1-dd94-4151-9b4f-6392155a2023-metrics-tls\") pod \"dns-default-vfn4g\" (UID: \"6c1405e1-dd94-4151-9b4f-6392155a2023\") " pod="openshift-dns/dns-default-vfn4g" Oct 30 18:33:53.573718 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.573209 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90488e60-cb70-4afa-a54b-6f333d345e84-metrics-certs\") pod \"network-metrics-daemon-dtppz\" (UID: \"90488e60-cb70-4afa-a54b-6f333d345e84\") " pod="openshift-multus/network-metrics-daemon-dtppz" Oct 30 18:33:53.573718 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.573320 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-958spucjgsqmv\"" Oct 30 18:33:53.573718 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.573437 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d2baf406-5501-4717-b5e1-d145edfe9ddc-cni-binary-copy\") pod \"multus-68r7k\" (UID: \"d2baf406-5501-4717-b5e1-d145edfe9ddc\") " pod="openshift-multus/multus-68r7k" Oct 30 18:33:53.573718 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.573520 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxzzq\" (UniqueName: \"kubernetes.io/projected/5b826abe-3f79-43ab-8813-2406d5c4dae9-kube-api-access-kxzzq\") pod \"multus-additional-cni-plugins-nl4m8\" (UID: \"5b826abe-3f79-43ab-8813-2406d5c4dae9\") " pod="openshift-multus/multus-additional-cni-plugins-nl4m8" Oct 30 18:33:53.573718 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.573536 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b4961ad6-2fc3-4c01-a565-6f66068d2958-agent-certs\") pod \"konnectivity-agent-4g8sn\" (UID: \"b4961ad6-2fc3-4c01-a565-6f66068d2958\") " pod="kube-system/konnectivity-agent-4g8sn" Oct 30 18:33:53.573718 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.573551 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e91ad8e0-05ce-420f-9594-445f4d48739c-etc-selinux\") pod \"aws-ebs-csi-driver-node-qtz4g\" (UID: \"e91ad8e0-05ce-420f-9594-445f4d48739c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtz4g" Oct 30 18:33:53.573718 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.573580 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pf82\" (UniqueName: \"kubernetes.io/projected/0a1badb7-be53-4b7f-933f-23e2599ab5ba-kube-api-access-5pf82\") pod \"insights-runtime-extractor-w4clh\" (UID: \"0a1badb7-be53-4b7f-933f-23e2599ab5ba\") " pod="openshift-insights/insights-runtime-extractor-w4clh" Oct 30 18:33:53.573718 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.573586 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 18:33:53.573718 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.573603 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbvzt\" (UniqueName: \"kubernetes.io/projected/91716cdd-99fb-4140-a52c-6d357ac74fb1-kube-api-access-bbvzt\") pod \"ingress-canary-bz2mv\" (UID: \"91716cdd-99fb-4140-a52c-6d357ac74fb1\") " pod="openshift-ingress-canary/ingress-canary-bz2mv" Oct 30 18:33:53.573718 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.573627 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/81f52134-6a09-431f-9f80-412ae7bbead8-run-systemd\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:33:53.573718 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.573650 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rncj\" (UniqueName: \"kubernetes.io/projected/b0f929fc-a9ea-4935-9fe7-770c5e7d4de5-kube-api-access-4rncj\") pod \"node-ca-qrdc6\" (UID: \"b0f929fc-a9ea-4935-9fe7-770c5e7d4de5\") " pod="openshift-image-registry/node-ca-qrdc6" Oct 30 18:33:53.573718 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.573672 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c1405e1-dd94-4151-9b4f-6392155a2023-tmp-dir\") pod \"dns-default-vfn4g\" (UID: \"6c1405e1-dd94-4151-9b4f-6392155a2023\") " pod="openshift-dns/dns-default-vfn4g" Oct 30 18:33:53.575370 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.573782 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/81f52134-6a09-431f-9f80-412ae7bbead8-etc-openvswitch\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:33:53.575370 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.573811 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/81f52134-6a09-431f-9f80-412ae7bbead8-run-openvswitch\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:33:53.575370 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.573859 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdz5l\" (UniqueName: \"kubernetes.io/projected/81f52134-6a09-431f-9f80-412ae7bbead8-kube-api-access-rdz5l\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:33:53.575370 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.573885 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d2baf406-5501-4717-b5e1-d145edfe9ddc-system-cni-dir\") pod \"multus-68r7k\" (UID: \"d2baf406-5501-4717-b5e1-d145edfe9ddc\") " pod="openshift-multus/multus-68r7k" Oct 30 18:33:53.575370 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.573902 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj8gn\" (UniqueName: \"kubernetes.io/projected/05cad8b6-56f5-4bd1-b12d-b2ae456b7027-kube-api-access-vj8gn\") pod \"node-resolver-bk5br\" (UID: \"05cad8b6-56f5-4bd1-b12d-b2ae456b7027\") " pod="openshift-dns/node-resolver-bk5br" Oct 30 18:33:53.575370 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.573916 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/eef45986-9475-4e3e-a3f9-fdb682474ff3-run\") pod \"tuned-rpzvk\" (UID: \"eef45986-9475-4e3e-a3f9-fdb682474ff3\") " pod="openshift-cluster-node-tuning-operator/tuned-rpzvk" Oct 30 18:33:53.575370 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.573961 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knxck\" (UniqueName: \"kubernetes.io/projected/eef45986-9475-4e3e-a3f9-fdb682474ff3-kube-api-access-knxck\") pod \"tuned-rpzvk\" (UID: \"eef45986-9475-4e3e-a3f9-fdb682474ff3\") " pod="openshift-cluster-node-tuning-operator/tuned-rpzvk" Oct 30 18:33:53.575370 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.574037 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c1405e1-dd94-4151-9b4f-6392155a2023-tmp-dir\") pod \"dns-default-vfn4g\" (UID: \"6c1405e1-dd94-4151-9b4f-6392155a2023\") " pod="openshift-dns/dns-default-vfn4g" Oct 30 18:33:53.575370 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.574098 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/81f52134-6a09-431f-9f80-412ae7bbead8-ovnkube-config\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:33:53.575370 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.574119 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d2baf406-5501-4717-b5e1-d145edfe9ddc-multus-cni-dir\") pod \"multus-68r7k\" (UID: \"d2baf406-5501-4717-b5e1-d145edfe9ddc\") " pod="openshift-multus/multus-68r7k" Oct 30 18:33:53.575370 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.574134 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d2baf406-5501-4717-b5e1-d145edfe9ddc-cnibin\") pod \"multus-68r7k\" (UID: \"d2baf406-5501-4717-b5e1-d145edfe9ddc\") " pod="openshift-multus/multus-68r7k" Oct 30 18:33:53.575370 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.574187 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d2baf406-5501-4717-b5e1-d145edfe9ddc-host-run-k8s-cni-cncf-io\") pod \"multus-68r7k\" (UID: \"d2baf406-5501-4717-b5e1-d145edfe9ddc\") " pod="openshift-multus/multus-68r7k" Oct 30 18:33:53.575370 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.574223 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/312dae04-dfda-41da-90ca-36b3b44a12ea-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-8698b\" (UID: \"312dae04-dfda-41da-90ca-36b3b44a12ea\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8698b" Oct 30 18:33:53.575370 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.574255 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l927s\" (UniqueName: \"kubernetes.io/projected/ef86d9fa-a344-4886-b80c-89ce84f418ea-kube-api-access-l927s\") pod \"iptables-alerter-b876b\" (UID: \"ef86d9fa-a344-4886-b80c-89ce84f418ea\") " pod="openshift-network-operator/iptables-alerter-b876b" Oct 30 18:33:53.575370 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.574312 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/81f52134-6a09-431f-9f80-412ae7bbead8-ovnkube-config\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:33:53.575370 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.574326 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5b826abe-3f79-43ab-8813-2406d5c4dae9-cni-binary-copy\") pod \"multus-additional-cni-plugins-nl4m8\" (UID: \"5b826abe-3f79-43ab-8813-2406d5c4dae9\") " pod="openshift-multus/multus-additional-cni-plugins-nl4m8" Oct 30 18:33:53.575370 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.574358 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e91ad8e0-05ce-420f-9594-445f4d48739c-sys-fs\") pod \"aws-ebs-csi-driver-node-qtz4g\" (UID: \"e91ad8e0-05ce-420f-9594-445f4d48739c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtz4g" Oct 30 18:33:53.575370 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.574384 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/91716cdd-99fb-4140-a52c-6d357ac74fb1-cert\") pod \"ingress-canary-bz2mv\" (UID: \"91716cdd-99fb-4140-a52c-6d357ac74fb1\") " pod="openshift-ingress-canary/ingress-canary-bz2mv" Oct 30 18:33:53.575370 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.574412 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/312dae04-dfda-41da-90ca-36b3b44a12ea-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-8698b\" (UID: \"312dae04-dfda-41da-90ca-36b3b44a12ea\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8698b" Oct 30 18:33:53.575370 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.574446 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/312dae04-dfda-41da-90ca-36b3b44a12ea-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-8698b\" (UID: \"312dae04-dfda-41da-90ca-36b3b44a12ea\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8698b" Oct 30 18:33:53.575370 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.574483 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/312dae04-dfda-41da-90ca-36b3b44a12ea-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-8698b\" (UID: \"312dae04-dfda-41da-90ca-36b3b44a12ea\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8698b" Oct 30 18:33:53.575370 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.574473 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztd9j\" (UniqueName: \"kubernetes.io/projected/fb66d280-cf24-4549-8fdb-89f16b6080c1-kube-api-access-ztd9j\") pod \"network-check-target-26f57\" (UID: \"fb66d280-cf24-4549-8fdb-89f16b6080c1\") " pod="openshift-network-diagnostics/network-check-target-26f57" Oct 30 18:33:53.575370 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.574517 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eef45986-9475-4e3e-a3f9-fdb682474ff3-etc-kubernetes\") pod \"tuned-rpzvk\" (UID: \"eef45986-9475-4e3e-a3f9-fdb682474ff3\") " pod="openshift-cluster-node-tuning-operator/tuned-rpzvk" Oct 30 18:33:53.575370 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.574541 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/05cad8b6-56f5-4bd1-b12d-b2ae456b7027-hosts-file\") pod \"node-resolver-bk5br\" (UID: \"05cad8b6-56f5-4bd1-b12d-b2ae456b7027\") " pod="openshift-dns/node-resolver-bk5br" Oct 30 18:33:53.575370 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.574572 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5b826abe-3f79-43ab-8813-2406d5c4dae9-cni-binary-copy\") pod \"multus-additional-cni-plugins-nl4m8\" (UID: \"5b826abe-3f79-43ab-8813-2406d5c4dae9\") " pod="openshift-multus/multus-additional-cni-plugins-nl4m8" Oct 30 18:33:53.575370 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.574618 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/312dae04-dfda-41da-90ca-36b3b44a12ea-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-8698b\" (UID: \"312dae04-dfda-41da-90ca-36b3b44a12ea\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8698b" Oct 30 18:33:53.575370 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.574628 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/91716cdd-99fb-4140-a52c-6d357ac74fb1-cert\") pod \"ingress-canary-bz2mv\" (UID: \"91716cdd-99fb-4140-a52c-6d357ac74fb1\") " pod="openshift-ingress-canary/ingress-canary-bz2mv" Oct 30 18:33:53.575370 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.574670 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/312dae04-dfda-41da-90ca-36b3b44a12ea-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-8698b\" (UID: \"312dae04-dfda-41da-90ca-36b3b44a12ea\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8698b" Oct 30 18:33:53.575370 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.574704 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ef86d9fa-a344-4886-b80c-89ce84f418ea-iptables-alerter-script\") pod \"iptables-alerter-b876b\" (UID: \"ef86d9fa-a344-4886-b80c-89ce84f418ea\") " pod="openshift-network-operator/iptables-alerter-b876b" Oct 30 18:33:53.575370 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.574738 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b826abe-3f79-43ab-8813-2406d5c4dae9-system-cni-dir\") pod \"multus-additional-cni-plugins-nl4m8\" (UID: \"5b826abe-3f79-43ab-8813-2406d5c4dae9\") " pod="openshift-multus/multus-additional-cni-plugins-nl4m8" Oct 30 18:33:53.575370 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.574767 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5b826abe-3f79-43ab-8813-2406d5c4dae9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nl4m8\" (UID: \"5b826abe-3f79-43ab-8813-2406d5c4dae9\") " pod="openshift-multus/multus-additional-cni-plugins-nl4m8" Oct 30 18:33:53.575370 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.574817 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e91ad8e0-05ce-420f-9594-445f4d48739c-device-dir\") pod \"aws-ebs-csi-driver-node-qtz4g\" (UID: \"e91ad8e0-05ce-420f-9594-445f4d48739c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtz4g" Oct 30 18:33:53.575370 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.574835 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e163025a-b19f-44e2-9058-b65f31bfbcc4-kubelet-config\") pod \"global-pull-secret-syncer-zmdjg\" (UID: \"e163025a-b19f-44e2-9058-b65f31bfbcc4\") " pod="kube-system/global-pull-secret-syncer-zmdjg" Oct 30 18:33:53.575370 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.574838 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ef86d9fa-a344-4886-b80c-89ce84f418ea-iptables-alerter-script\") pod \"iptables-alerter-b876b\" (UID: \"ef86d9fa-a344-4886-b80c-89ce84f418ea\") " pod="openshift-network-operator/iptables-alerter-b876b" Oct 30 18:33:53.575370 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.574857 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0a1badb7-be53-4b7f-933f-23e2599ab5ba-data-volume\") pod \"insights-runtime-extractor-w4clh\" (UID: \"0a1badb7-be53-4b7f-933f-23e2599ab5ba\") " pod="openshift-insights/insights-runtime-extractor-w4clh" Oct 30 18:33:53.575370 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.574903 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a1badb7-be53-4b7f-933f-23e2599ab5ba-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w4clh\" (UID: \"0a1badb7-be53-4b7f-933f-23e2599ab5ba\") " pod="openshift-insights/insights-runtime-extractor-w4clh" Oct 30 18:33:53.575370 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.574918 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6c1405e1-dd94-4151-9b4f-6392155a2023-config-volume\") pod \"dns-default-vfn4g\" (UID: \"6c1405e1-dd94-4151-9b4f-6392155a2023\") " pod="openshift-dns/dns-default-vfn4g" Oct 30 18:33:53.575370 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.574925 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5b826abe-3f79-43ab-8813-2406d5c4dae9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nl4m8\" (UID: \"5b826abe-3f79-43ab-8813-2406d5c4dae9\") " pod="openshift-multus/multus-additional-cni-plugins-nl4m8" Oct 30 18:33:53.575370 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.574932 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/81f52134-6a09-431f-9f80-412ae7bbead8-host-slash\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:33:53.575370 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.574966 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81f52134-6a09-431f-9f80-412ae7bbead8-host-run-ovn-kubernetes\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:33:53.575370 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.575060 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/81f52134-6a09-431f-9f80-412ae7bbead8-host-cni-netd\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:33:53.575370 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.575080 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0a1badb7-be53-4b7f-933f-23e2599ab5ba-data-volume\") pod \"insights-runtime-extractor-w4clh\" (UID: \"0a1badb7-be53-4b7f-933f-23e2599ab5ba\") " pod="openshift-insights/insights-runtime-extractor-w4clh" Oct 30 18:33:53.575370 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.575103 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6c1405e1-dd94-4151-9b4f-6392155a2023-config-volume\") pod \"dns-default-vfn4g\" (UID: \"6c1405e1-dd94-4151-9b4f-6392155a2023\") " pod="openshift-dns/dns-default-vfn4g" Oct 30 18:33:53.575370 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.575137 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a1badb7-be53-4b7f-933f-23e2599ab5ba-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-w4clh\" (UID: \"0a1badb7-be53-4b7f-933f-23e2599ab5ba\") " pod="openshift-insights/insights-runtime-extractor-w4clh" Oct 30 18:33:53.577189 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.577168 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 18:33:53.577548 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.577526 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 18:33:53.578191 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.578174 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 18:33:53.580639 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.580617 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4s85\" (UniqueName: \"kubernetes.io/projected/d2baf406-5501-4717-b5e1-d145edfe9ddc-kube-api-access-m4s85\") pod \"multus-68r7k\" (UID: \"d2baf406-5501-4717-b5e1-d145edfe9ddc\") " pod="openshift-multus/multus-68r7k" Oct 30 18:33:53.580700 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.580612 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzfgl\" (UniqueName: \"kubernetes.io/projected/90488e60-cb70-4afa-a54b-6f333d345e84-kube-api-access-xzfgl\") pod \"network-metrics-daemon-dtppz\" (UID: \"90488e60-cb70-4afa-a54b-6f333d345e84\") " pod="openshift-multus/network-metrics-daemon-dtppz" Oct 30 18:33:53.597334 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.597314 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb2bf\" (UniqueName: \"kubernetes.io/projected/e91ad8e0-05ce-420f-9594-445f4d48739c-kube-api-access-nb2bf\") pod \"aws-ebs-csi-driver-node-qtz4g\" (UID: \"e91ad8e0-05ce-420f-9594-445f4d48739c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtz4g" Oct 30 18:33:53.597430 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.597336 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z8lm\" (UniqueName: \"kubernetes.io/projected/6c1405e1-dd94-4151-9b4f-6392155a2023-kube-api-access-6z8lm\") pod \"dns-default-vfn4g\" (UID: \"6c1405e1-dd94-4151-9b4f-6392155a2023\") " pod="openshift-dns/dns-default-vfn4g" Oct 30 18:33:53.597499 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.597432 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8422\" (UniqueName: \"kubernetes.io/projected/312dae04-dfda-41da-90ca-36b3b44a12ea-kube-api-access-z8422\") pod \"openshift-state-metrics-7bd7489957-8698b\" (UID: \"312dae04-dfda-41da-90ca-36b3b44a12ea\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8698b" Oct 30 18:33:53.599562 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.599544 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pf82\" (UniqueName: \"kubernetes.io/projected/0a1badb7-be53-4b7f-933f-23e2599ab5ba-kube-api-access-5pf82\") pod \"insights-runtime-extractor-w4clh\" (UID: \"0a1badb7-be53-4b7f-933f-23e2599ab5ba\") " pod="openshift-insights/insights-runtime-extractor-w4clh" Oct 30 18:33:53.599777 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.599762 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdz5l\" (UniqueName: \"kubernetes.io/projected/81f52134-6a09-431f-9f80-412ae7bbead8-kube-api-access-rdz5l\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:33:53.599953 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.599935 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztd9j\" (UniqueName: \"kubernetes.io/projected/fb66d280-cf24-4549-8fdb-89f16b6080c1-kube-api-access-ztd9j\") pod \"network-check-target-26f57\" (UID: \"fb66d280-cf24-4549-8fdb-89f16b6080c1\") " pod="openshift-network-diagnostics/network-check-target-26f57" Oct 30 18:33:53.600759 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.600745 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbvzt\" (UniqueName: \"kubernetes.io/projected/91716cdd-99fb-4140-a52c-6d357ac74fb1-kube-api-access-bbvzt\") pod \"ingress-canary-bz2mv\" (UID: \"91716cdd-99fb-4140-a52c-6d357ac74fb1\") " pod="openshift-ingress-canary/ingress-canary-bz2mv" Oct 30 18:33:53.602500 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.602479 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knxck\" (UniqueName: \"kubernetes.io/projected/eef45986-9475-4e3e-a3f9-fdb682474ff3-kube-api-access-knxck\") pod \"tuned-rpzvk\" (UID: \"eef45986-9475-4e3e-a3f9-fdb682474ff3\") " pod="openshift-cluster-node-tuning-operator/tuned-rpzvk" Oct 30 18:33:53.602718 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.602705 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj8gn\" (UniqueName: \"kubernetes.io/projected/05cad8b6-56f5-4bd1-b12d-b2ae456b7027-kube-api-access-vj8gn\") pod \"node-resolver-bk5br\" (UID: \"05cad8b6-56f5-4bd1-b12d-b2ae456b7027\") " pod="openshift-dns/node-resolver-bk5br" Oct 30 18:33:53.602778 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.602756 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rncj\" (UniqueName: \"kubernetes.io/projected/b0f929fc-a9ea-4935-9fe7-770c5e7d4de5-kube-api-access-4rncj\") pod \"node-ca-qrdc6\" (UID: \"b0f929fc-a9ea-4935-9fe7-770c5e7d4de5\") " pod="openshift-image-registry/node-ca-qrdc6" Oct 30 18:33:53.604198 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.604180 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l927s\" (UniqueName: \"kubernetes.io/projected/ef86d9fa-a344-4886-b80c-89ce84f418ea-kube-api-access-l927s\") pod \"iptables-alerter-b876b\" (UID: \"ef86d9fa-a344-4886-b80c-89ce84f418ea\") " pod="openshift-network-operator/iptables-alerter-b876b" Oct 30 18:33:53.604198 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.604193 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxzzq\" (UniqueName: \"kubernetes.io/projected/5b826abe-3f79-43ab-8813-2406d5c4dae9-kube-api-access-kxzzq\") pod \"multus-additional-cni-plugins-nl4m8\" (UID: \"5b826abe-3f79-43ab-8813-2406d5c4dae9\") " pod="openshift-multus/multus-additional-cni-plugins-nl4m8" Oct 30 18:33:53.650355 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.650332 6960 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-163.us-east-2.compute.internal" Oct 30 18:33:53.654953 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.654933 6960 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 18:33:53.661513 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.661495 6960 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 18:33:53.661603 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:33:53.661539 6960 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-140-163.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-163.us-east-2.compute.internal" Oct 30 18:33:53.675659 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.675638 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8f76499e-97ad-4c03-8a72-a62f65619596-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:53.675778 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.675667 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/eef45986-9475-4e3e-a3f9-fdb682474ff3-run\") pod \"tuned-rpzvk\" (UID: \"eef45986-9475-4e3e-a3f9-fdb682474ff3\") " pod="openshift-cluster-node-tuning-operator/tuned-rpzvk" Oct 30 18:33:53.675778 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.675706 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/433ed181-b45e-45c0-a593-9cb0b6c09348-sys\") pod \"node-exporter-hh862\" (UID: \"433ed181-b45e-45c0-a593-9cb0b6c09348\") " pod="openshift-monitoring/node-exporter-hh862" Oct 30 18:33:53.675778 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.675723 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8f76499e-97ad-4c03-8a72-a62f65619596-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:53.675778 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.675729 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2069a85a-44e7-4c98-9f47-7fad4416c15b-client-ca-bundle\") pod \"metrics-server-b6d7bcd58-z2xvj\" (UID: \"2069a85a-44e7-4c98-9f47-7fad4416c15b\") " pod="openshift-monitoring/metrics-server-b6d7bcd58-z2xvj" Oct 30 18:33:53.675778 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.675765 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/eef45986-9475-4e3e-a3f9-fdb682474ff3-run\") pod \"tuned-rpzvk\" (UID: \"eef45986-9475-4e3e-a3f9-fdb682474ff3\") " pod="openshift-cluster-node-tuning-operator/tuned-rpzvk" Oct 30 18:33:53.675946 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.675811 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:33:53.675946 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.675851 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/158becf0-d6cf-4917-8329-4f64ae5a4ca4-secret-thanos-querier-tls\") pod \"thanos-querier-7c6f76d6d-k2h7h\" (UID: \"158becf0-d6cf-4917-8329-4f64ae5a4ca4\") " pod="openshift-monitoring/thanos-querier-7c6f76d6d-k2h7h" Oct 30 18:33:53.675946 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.675920 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/2069a85a-44e7-4c98-9f47-7fad4416c15b-audit-log\") pod \"metrics-server-b6d7bcd58-z2xvj\" (UID: \"2069a85a-44e7-4c98-9f47-7fad4416c15b\") " pod="openshift-monitoring/metrics-server-b6d7bcd58-z2xvj" Oct 30 18:33:53.676090 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.675952 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/05cad8b6-56f5-4bd1-b12d-b2ae456b7027-hosts-file\") pod \"node-resolver-bk5br\" (UID: \"05cad8b6-56f5-4bd1-b12d-b2ae456b7027\") " pod="openshift-dns/node-resolver-bk5br" Oct 30 18:33:53.676090 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.675977 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e163025a-b19f-44e2-9058-b65f31bfbcc4-kubelet-config\") pod \"global-pull-secret-syncer-zmdjg\" (UID: \"e163025a-b19f-44e2-9058-b65f31bfbcc4\") " pod="kube-system/global-pull-secret-syncer-zmdjg" Oct 30 18:33:53.676219 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.676103 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/433ed181-b45e-45c0-a593-9cb0b6c09348-node-exporter-textfile\") pod \"node-exporter-hh862\" (UID: \"433ed181-b45e-45c0-a593-9cb0b6c09348\") " pod="openshift-monitoring/node-exporter-hh862" Oct 30 18:33:53.676219 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.676113 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:33:53.676219 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.676158 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/158becf0-d6cf-4917-8329-4f64ae5a4ca4-secret-thanos-querier-tls\") pod \"thanos-querier-7c6f76d6d-k2h7h\" (UID: \"158becf0-d6cf-4917-8329-4f64ae5a4ca4\") " pod="openshift-monitoring/thanos-querier-7c6f76d6d-k2h7h" Oct 30 18:33:53.676219 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.676178 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8f76499e-97ad-4c03-8a72-a62f65619596-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:53.676219 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.676194 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2069a85a-44e7-4c98-9f47-7fad4416c15b-client-ca-bundle\") pod \"metrics-server-b6d7bcd58-z2xvj\" (UID: \"2069a85a-44e7-4c98-9f47-7fad4416c15b\") " pod="openshift-monitoring/metrics-server-b6d7bcd58-z2xvj" Oct 30 18:33:53.676219 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.676201 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/05cad8b6-56f5-4bd1-b12d-b2ae456b7027-hosts-file\") pod \"node-resolver-bk5br\" (UID: \"05cad8b6-56f5-4bd1-b12d-b2ae456b7027\") " pod="openshift-dns/node-resolver-bk5br" Oct 30 18:33:53.676219 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.676215 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e163025a-b19f-44e2-9058-b65f31bfbcc4-kubelet-config\") pod \"global-pull-secret-syncer-zmdjg\" (UID: \"e163025a-b19f-44e2-9058-b65f31bfbcc4\") " pod="kube-system/global-pull-secret-syncer-zmdjg" Oct 30 18:33:53.676464 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.676268 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/81f52134-6a09-431f-9f80-412ae7bbead8-host-cni-netd\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:33:53.676464 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.676274 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/2069a85a-44e7-4c98-9f47-7fad4416c15b-audit-log\") pod \"metrics-server-b6d7bcd58-z2xvj\" (UID: \"2069a85a-44e7-4c98-9f47-7fad4416c15b\") " pod="openshift-monitoring/metrics-server-b6d7bcd58-z2xvj" Oct 30 18:33:53.676464 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.676312 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/433ed181-b45e-45c0-a593-9cb0b6c09348-node-exporter-textfile\") pod \"node-exporter-hh862\" (UID: \"433ed181-b45e-45c0-a593-9cb0b6c09348\") " pod="openshift-monitoring/node-exporter-hh862" Oct 30 18:33:53.676464 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.676366 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/4f8841c0-4cf3-45d9-9301-957d4fffe1ce-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-7gk7g\" (UID: \"4f8841c0-4cf3-45d9-9301-957d4fffe1ce\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7gk7g" Oct 30 18:33:53.676464 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.676394 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/81f52134-6a09-431f-9f80-412ae7bbead8-host-cni-netd\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:33:53.676464 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.676430 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:33:53.676464 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.676448 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/4f8841c0-4cf3-45d9-9301-957d4fffe1ce-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-7gk7g\" (UID: \"4f8841c0-4cf3-45d9-9301-957d4fffe1ce\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7gk7g" Oct 30 18:33:53.676739 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.676484 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/158becf0-d6cf-4917-8329-4f64ae5a4ca4-metrics-client-ca\") pod \"thanos-querier-7c6f76d6d-k2h7h\" (UID: \"158becf0-d6cf-4917-8329-4f64ae5a4ca4\") " pod="openshift-monitoring/thanos-querier-7c6f76d6d-k2h7h" Oct 30 18:33:53.676739 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.676533 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8f76499e-97ad-4c03-8a72-a62f65619596-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:53.676739 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.676585 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81f52134-6a09-431f-9f80-412ae7bbead8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:33:53.676739 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.676615 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d2baf406-5501-4717-b5e1-d145edfe9ddc-host-run-multus-certs\") pod \"multus-68r7k\" (UID: \"d2baf406-5501-4717-b5e1-d145edfe9ddc\") " pod="openshift-multus/multus-68r7k" Oct 30 18:33:53.676739 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.676640 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9nwr\" (UniqueName: \"kubernetes.io/projected/4f8841c0-4cf3-45d9-9301-957d4fffe1ce-kube-api-access-x9nwr\") pod \"kube-state-metrics-5dc8d7d89b-7gk7g\" (UID: \"4f8841c0-4cf3-45d9-9301-957d4fffe1ce\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7gk7g" Oct 30 18:33:53.676739 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.676664 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8f76499e-97ad-4c03-8a72-a62f65619596-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:53.676739 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.676687 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b0f929fc-a9ea-4935-9fe7-770c5e7d4de5-host\") pod \"node-ca-qrdc6\" (UID: \"b0f929fc-a9ea-4935-9fe7-770c5e7d4de5\") " pod="openshift-image-registry/node-ca-qrdc6" Oct 30 18:33:53.676739 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.676699 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/158becf0-d6cf-4917-8329-4f64ae5a4ca4-metrics-client-ca\") pod \"thanos-querier-7c6f76d6d-k2h7h\" (UID: \"158becf0-d6cf-4917-8329-4f64ae5a4ca4\") " pod="openshift-monitoring/thanos-querier-7c6f76d6d-k2h7h" Oct 30 18:33:53.676739 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.676710 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8f76499e-97ad-4c03-8a72-a62f65619596-web-config\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:53.677021 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.676747 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/2069a85a-44e7-4c98-9f47-7fad4416c15b-secret-metrics-server-tls\") pod \"metrics-server-b6d7bcd58-z2xvj\" (UID: \"2069a85a-44e7-4c98-9f47-7fad4416c15b\") " pod="openshift-monitoring/metrics-server-b6d7bcd58-z2xvj" Oct 30 18:33:53.677021 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.676775 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/eef45986-9475-4e3e-a3f9-fdb682474ff3-etc-sysctl-conf\") pod \"tuned-rpzvk\" (UID: \"eef45986-9475-4e3e-a3f9-fdb682474ff3\") " pod="openshift-cluster-node-tuning-operator/tuned-rpzvk" Oct 30 18:33:53.677021 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.676807 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:33:53.677021 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.676803 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:33:53.677021 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.676832 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e91ad8e0-05ce-420f-9594-445f4d48739c-socket-dir\") pod \"aws-ebs-csi-driver-node-qtz4g\" (UID: \"e91ad8e0-05ce-420f-9594-445f4d48739c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtz4g" Oct 30 18:33:53.677021 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.676861 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e91ad8e0-05ce-420f-9594-445f4d48739c-registration-dir\") pod \"aws-ebs-csi-driver-node-qtz4g\" (UID: \"e91ad8e0-05ce-420f-9594-445f4d48739c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtz4g" Oct 30 18:33:53.677021 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.676886 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0a1badb7-be53-4b7f-933f-23e2599ab5ba-crio-socket\") pod \"insights-runtime-extractor-w4clh\" (UID: \"0a1badb7-be53-4b7f-933f-23e2599ab5ba\") " pod="openshift-insights/insights-runtime-extractor-w4clh" Oct 30 18:33:53.677021 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.676916 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4f8841c0-4cf3-45d9-9301-957d4fffe1ce-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-7gk7g\" (UID: \"4f8841c0-4cf3-45d9-9301-957d4fffe1ce\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7gk7g" Oct 30 18:33:53.677021 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.676929 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8f76499e-97ad-4c03-8a72-a62f65619596-web-config\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:53.677021 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.676942 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zndl\" (UniqueName: \"kubernetes.io/projected/158becf0-d6cf-4917-8329-4f64ae5a4ca4-kube-api-access-4zndl\") pod \"thanos-querier-7c6f76d6d-k2h7h\" (UID: \"158becf0-d6cf-4917-8329-4f64ae5a4ca4\") " pod="openshift-monitoring/thanos-querier-7c6f76d6d-k2h7h" Oct 30 18:33:53.677021 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.676956 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/eef45986-9475-4e3e-a3f9-fdb682474ff3-etc-sysctl-conf\") pod \"tuned-rpzvk\" (UID: \"eef45986-9475-4e3e-a3f9-fdb682474ff3\") " pod="openshift-cluster-node-tuning-operator/tuned-rpzvk" Oct 30 18:33:53.677021 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.676969 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eef45986-9475-4e3e-a3f9-fdb682474ff3-sys\") pod \"tuned-rpzvk\" (UID: \"eef45986-9475-4e3e-a3f9-fdb682474ff3\") " pod="openshift-cluster-node-tuning-operator/tuned-rpzvk" Oct 30 18:33:53.677021 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.676994 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/81f52134-6a09-431f-9f80-412ae7bbead8-host-kubelet\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:33:53.677021 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.677017 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/81f52134-6a09-431f-9f80-412ae7bbead8-systemd-units\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:33:53.677507 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.677057 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:33:53.677507 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.677081 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-config-out\") pod \"alertmanager-main-0\" (UID: \"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:33:53.677507 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.677105 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5b826abe-3f79-43ab-8813-2406d5c4dae9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nl4m8\" (UID: \"5b826abe-3f79-43ab-8813-2406d5c4dae9\") " pod="openshift-multus/multus-additional-cni-plugins-nl4m8" Oct 30 18:33:53.677507 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.677119 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/2069a85a-44e7-4c98-9f47-7fad4416c15b-secret-metrics-server-tls\") pod \"metrics-server-b6d7bcd58-z2xvj\" (UID: \"2069a85a-44e7-4c98-9f47-7fad4416c15b\") " pod="openshift-monitoring/metrics-server-b6d7bcd58-z2xvj" Oct 30 18:33:53.677507 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.677118 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8f76499e-97ad-4c03-8a72-a62f65619596-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:53.677507 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.677136 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/81f52134-6a09-431f-9f80-412ae7bbead8-host-cni-bin\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:33:53.677507 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.677134 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81f52134-6a09-431f-9f80-412ae7bbead8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:33:53.677507 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.677181 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b0f929fc-a9ea-4935-9fe7-770c5e7d4de5-host\") pod \"node-ca-qrdc6\" (UID: \"b0f929fc-a9ea-4935-9fe7-770c5e7d4de5\") " pod="openshift-image-registry/node-ca-qrdc6" Oct 30 18:33:53.677507 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.677192 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4f8841c0-4cf3-45d9-9301-957d4fffe1ce-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-7gk7g\" (UID: \"4f8841c0-4cf3-45d9-9301-957d4fffe1ce\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7gk7g" Oct 30 18:33:53.677507 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.677217 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d2baf406-5501-4717-b5e1-d145edfe9ddc-host-run-multus-certs\") pod \"multus-68r7k\" (UID: \"d2baf406-5501-4717-b5e1-d145edfe9ddc\") " pod="openshift-multus/multus-68r7k" Oct 30 18:33:53.677507 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.677211 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/81f52134-6a09-431f-9f80-412ae7bbead8-host-cni-bin\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:33:53.677507 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.677221 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8f76499e-97ad-4c03-8a72-a62f65619596-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:53.677507 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.677260 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e91ad8e0-05ce-420f-9594-445f4d48739c-socket-dir\") pod \"aws-ebs-csi-driver-node-qtz4g\" (UID: \"e91ad8e0-05ce-420f-9594-445f4d48739c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtz4g" Oct 30 18:33:53.677507 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.677279 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e91ad8e0-05ce-420f-9594-445f4d48739c-registration-dir\") pod \"aws-ebs-csi-driver-node-qtz4g\" (UID: \"e91ad8e0-05ce-420f-9594-445f4d48739c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtz4g" Oct 30 18:33:53.677507 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.677290 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0a1badb7-be53-4b7f-933f-23e2599ab5ba-crio-socket\") pod \"insights-runtime-extractor-w4clh\" (UID: \"0a1badb7-be53-4b7f-933f-23e2599ab5ba\") " pod="openshift-insights/insights-runtime-extractor-w4clh" Oct 30 18:33:53.677507 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.677325 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/81f52134-6a09-431f-9f80-412ae7bbead8-systemd-units\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:33:53.677507 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.677442 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8f76499e-97ad-4c03-8a72-a62f65619596-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:53.677507 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.677439 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4f8841c0-4cf3-45d9-9301-957d4fffe1ce-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-7gk7g\" (UID: \"4f8841c0-4cf3-45d9-9301-957d4fffe1ce\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7gk7g" Oct 30 18:33:53.677507 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.677226 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eef45986-9475-4e3e-a3f9-fdb682474ff3-sys\") pod \"tuned-rpzvk\" (UID: \"eef45986-9475-4e3e-a3f9-fdb682474ff3\") " pod="openshift-cluster-node-tuning-operator/tuned-rpzvk" Oct 30 18:33:53.678058 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.677522 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/81f52134-6a09-431f-9f80-412ae7bbead8-host-kubelet\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:33:53.678058 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.677634 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:33:53.678058 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.677664 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4f8841c0-4cf3-45d9-9301-957d4fffe1ce-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-7gk7g\" (UID: \"4f8841c0-4cf3-45d9-9301-957d4fffe1ce\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7gk7g" Oct 30 18:33:53.678058 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.677666 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:33:53.678058 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.677643 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f76499e-97ad-4c03-8a72-a62f65619596-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:53.678058 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.677693 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-config-out\") pod \"alertmanager-main-0\" (UID: \"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:33:53.678058 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.677776 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8f76499e-97ad-4c03-8a72-a62f65619596-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:53.678058 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.677789 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5b826abe-3f79-43ab-8813-2406d5c4dae9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nl4m8\" (UID: \"5b826abe-3f79-43ab-8813-2406d5c4dae9\") " pod="openshift-multus/multus-additional-cni-plugins-nl4m8" Oct 30 18:33:53.678058 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.677807 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/2069a85a-44e7-4c98-9f47-7fad4416c15b-secret-metrics-server-client-certs\") pod \"metrics-server-b6d7bcd58-z2xvj\" (UID: \"2069a85a-44e7-4c98-9f47-7fad4416c15b\") " pod="openshift-monitoring/metrics-server-b6d7bcd58-z2xvj" Oct 30 18:33:53.678058 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.677830 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f76499e-97ad-4c03-8a72-a62f65619596-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:53.678058 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.677850 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eef45986-9475-4e3e-a3f9-fdb682474ff3-lib-modules\") pod \"tuned-rpzvk\" (UID: \"eef45986-9475-4e3e-a3f9-fdb682474ff3\") " pod="openshift-cluster-node-tuning-operator/tuned-rpzvk" Oct 30 18:33:53.678058 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.677902 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8f76499e-97ad-4c03-8a72-a62f65619596-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:53.678058 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.677939 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eef45986-9475-4e3e-a3f9-fdb682474ff3-host\") pod \"tuned-rpzvk\" (UID: \"eef45986-9475-4e3e-a3f9-fdb682474ff3\") " pod="openshift-cluster-node-tuning-operator/tuned-rpzvk" Oct 30 18:33:53.678058 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.677963 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/158becf0-d6cf-4917-8329-4f64ae5a4ca4-secret-grpc-tls\") pod \"thanos-querier-7c6f76d6d-k2h7h\" (UID: \"158becf0-d6cf-4917-8329-4f64ae5a4ca4\") " pod="openshift-monitoring/thanos-querier-7c6f76d6d-k2h7h" Oct 30 18:33:53.678058 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.678032 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/433ed181-b45e-45c0-a593-9cb0b6c09348-node-exporter-tls\") pod \"node-exporter-hh862\" (UID: \"433ed181-b45e-45c0-a593-9cb0b6c09348\") " pod="openshift-monitoring/node-exporter-hh862" Oct 30 18:33:53.678058 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.678037 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/2069a85a-44e7-4c98-9f47-7fad4416c15b-secret-metrics-server-client-certs\") pod \"metrics-server-b6d7bcd58-z2xvj\" (UID: \"2069a85a-44e7-4c98-9f47-7fad4416c15b\") " pod="openshift-monitoring/metrics-server-b6d7bcd58-z2xvj" Oct 30 18:33:53.678704 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.678070 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eef45986-9475-4e3e-a3f9-fdb682474ff3-lib-modules\") pod \"tuned-rpzvk\" (UID: \"eef45986-9475-4e3e-a3f9-fdb682474ff3\") " pod="openshift-cluster-node-tuning-operator/tuned-rpzvk" Oct 30 18:33:53.678704 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.678109 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/158becf0-d6cf-4917-8329-4f64ae5a4ca4-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7c6f76d6d-k2h7h\" (UID: \"158becf0-d6cf-4917-8329-4f64ae5a4ca4\") " pod="openshift-monitoring/thanos-querier-7c6f76d6d-k2h7h" Oct 30 18:33:53.678704 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.678124 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/eef45986-9475-4e3e-a3f9-fdb682474ff3-host\") pod \"tuned-rpzvk\" (UID: \"eef45986-9475-4e3e-a3f9-fdb682474ff3\") " pod="openshift-cluster-node-tuning-operator/tuned-rpzvk" Oct 30 18:33:53.678704 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.678224 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/158becf0-d6cf-4917-8329-4f64ae5a4ca4-secret-grpc-tls\") pod \"thanos-querier-7c6f76d6d-k2h7h\" (UID: \"158becf0-d6cf-4917-8329-4f64ae5a4ca4\") " pod="openshift-monitoring/thanos-querier-7c6f76d6d-k2h7h" Oct 30 18:33:53.678704 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.678251 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:33:53.678704 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.678345 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq2pp\" (UniqueName: \"kubernetes.io/projected/2069a85a-44e7-4c98-9f47-7fad4416c15b-kube-api-access-wq2pp\") pod \"metrics-server-b6d7bcd58-z2xvj\" (UID: \"2069a85a-44e7-4c98-9f47-7fad4416c15b\") " pod="openshift-monitoring/metrics-server-b6d7bcd58-z2xvj" Oct 30 18:33:53.678704 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.678349 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/433ed181-b45e-45c0-a593-9cb0b6c09348-node-exporter-tls\") pod \"node-exporter-hh862\" (UID: \"433ed181-b45e-45c0-a593-9cb0b6c09348\") " pod="openshift-monitoring/node-exporter-hh862" Oct 30 18:33:53.678704 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.678427 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/81f52134-6a09-431f-9f80-412ae7bbead8-host-run-netns\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:33:53.678704 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.678437 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/158becf0-d6cf-4917-8329-4f64ae5a4ca4-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7c6f76d6d-k2h7h\" (UID: \"158becf0-d6cf-4917-8329-4f64ae5a4ca4\") " pod="openshift-monitoring/thanos-querier-7c6f76d6d-k2h7h" Oct 30 18:33:53.678704 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.678478 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/81f52134-6a09-431f-9f80-412ae7bbead8-host-run-netns\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:33:53.678704 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.678487 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:33:53.678704 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.678516 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/eef45986-9475-4e3e-a3f9-fdb682474ff3-etc-sysconfig\") pod \"tuned-rpzvk\" (UID: \"eef45986-9475-4e3e-a3f9-fdb682474ff3\") " pod="openshift-cluster-node-tuning-operator/tuned-rpzvk" Oct 30 18:33:53.678704 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.678534 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/81f52134-6a09-431f-9f80-412ae7bbead8-run-systemd\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:33:53.678704 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.678540 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:33:53.678704 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.678548 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/433ed181-b45e-45c0-a593-9cb0b6c09348-node-exporter-accelerators-collector-config\") pod \"node-exporter-hh862\" (UID: \"433ed181-b45e-45c0-a593-9cb0b6c09348\") " pod="openshift-monitoring/node-exporter-hh862" Oct 30 18:33:53.678704 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.678564 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/81f52134-6a09-431f-9f80-412ae7bbead8-etc-openvswitch\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:33:53.678704 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.678574 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/eef45986-9475-4e3e-a3f9-fdb682474ff3-etc-sysconfig\") pod \"tuned-rpzvk\" (UID: \"eef45986-9475-4e3e-a3f9-fdb682474ff3\") " pod="openshift-cluster-node-tuning-operator/tuned-rpzvk" Oct 30 18:33:53.678704 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.678587 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/81f52134-6a09-431f-9f80-412ae7bbead8-run-systemd\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:33:53.678704 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.678588 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/81f52134-6a09-431f-9f80-412ae7bbead8-run-openvswitch\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:33:53.678704 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.678620 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/81f52134-6a09-431f-9f80-412ae7bbead8-etc-openvswitch\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:33:53.678704 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.678628 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d2baf406-5501-4717-b5e1-d145edfe9ddc-system-cni-dir\") pod \"multus-68r7k\" (UID: \"d2baf406-5501-4717-b5e1-d145edfe9ddc\") " pod="openshift-multus/multus-68r7k" Oct 30 18:33:53.678704 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.678654 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:33:53.678704 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.678663 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:33:53.678704 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.678678 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d2baf406-5501-4717-b5e1-d145edfe9ddc-multus-cni-dir\") pod \"multus-68r7k\" (UID: \"d2baf406-5501-4717-b5e1-d145edfe9ddc\") " pod="openshift-multus/multus-68r7k" Oct 30 18:33:53.678704 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.678666 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d2baf406-5501-4717-b5e1-d145edfe9ddc-system-cni-dir\") pod \"multus-68r7k\" (UID: \"d2baf406-5501-4717-b5e1-d145edfe9ddc\") " pod="openshift-multus/multus-68r7k" Oct 30 18:33:53.678704 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.678622 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/81f52134-6a09-431f-9f80-412ae7bbead8-run-openvswitch\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:33:53.678704 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.678713 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d2baf406-5501-4717-b5e1-d145edfe9ddc-cnibin\") pod \"multus-68r7k\" (UID: \"d2baf406-5501-4717-b5e1-d145edfe9ddc\") " pod="openshift-multus/multus-68r7k" Oct 30 18:33:53.679656 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.678733 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/433ed181-b45e-45c0-a593-9cb0b6c09348-node-exporter-accelerators-collector-config\") pod \"node-exporter-hh862\" (UID: \"433ed181-b45e-45c0-a593-9cb0b6c09348\") " pod="openshift-monitoring/node-exporter-hh862" Oct 30 18:33:53.679656 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.678741 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d2baf406-5501-4717-b5e1-d145edfe9ddc-multus-cni-dir\") pod \"multus-68r7k\" (UID: \"d2baf406-5501-4717-b5e1-d145edfe9ddc\") " pod="openshift-multus/multus-68r7k" Oct 30 18:33:53.679656 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.678735 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d2baf406-5501-4717-b5e1-d145edfe9ddc-host-run-k8s-cni-cncf-io\") pod \"multus-68r7k\" (UID: \"d2baf406-5501-4717-b5e1-d145edfe9ddc\") " pod="openshift-multus/multus-68r7k" Oct 30 18:33:53.679656 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.678764 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d2baf406-5501-4717-b5e1-d145edfe9ddc-host-run-k8s-cni-cncf-io\") pod \"multus-68r7k\" (UID: \"d2baf406-5501-4717-b5e1-d145edfe9ddc\") " pod="openshift-multus/multus-68r7k" Oct 30 18:33:53.679656 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.678795 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d2baf406-5501-4717-b5e1-d145edfe9ddc-cnibin\") pod \"multus-68r7k\" (UID: \"d2baf406-5501-4717-b5e1-d145edfe9ddc\") " pod="openshift-multus/multus-68r7k" Oct 30 18:33:53.679656 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.678807 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/433ed181-b45e-45c0-a593-9cb0b6c09348-root\") pod \"node-exporter-hh862\" (UID: \"433ed181-b45e-45c0-a593-9cb0b6c09348\") " pod="openshift-monitoring/node-exporter-hh862" Oct 30 18:33:53.679656 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.678828 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8f76499e-97ad-4c03-8a72-a62f65619596-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:53.679656 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.678846 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e91ad8e0-05ce-420f-9594-445f4d48739c-sys-fs\") pod \"aws-ebs-csi-driver-node-qtz4g\" (UID: \"e91ad8e0-05ce-420f-9594-445f4d48739c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtz4g" Oct 30 18:33:53.679656 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.678863 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-config-volume\") pod \"alertmanager-main-0\" (UID: \"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:33:53.679656 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.678884 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:33:53.679656 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.678886 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eef45986-9475-4e3e-a3f9-fdb682474ff3-etc-kubernetes\") pod \"tuned-rpzvk\" (UID: \"eef45986-9475-4e3e-a3f9-fdb682474ff3\") " pod="openshift-cluster-node-tuning-operator/tuned-rpzvk" Oct 30 18:33:53.679656 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.678924 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eef45986-9475-4e3e-a3f9-fdb682474ff3-etc-kubernetes\") pod \"tuned-rpzvk\" (UID: \"eef45986-9475-4e3e-a3f9-fdb682474ff3\") " pod="openshift-cluster-node-tuning-operator/tuned-rpzvk" Oct 30 18:33:53.679656 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.678931 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b826abe-3f79-43ab-8813-2406d5c4dae9-system-cni-dir\") pod \"multus-additional-cni-plugins-nl4m8\" (UID: \"5b826abe-3f79-43ab-8813-2406d5c4dae9\") " pod="openshift-multus/multus-additional-cni-plugins-nl4m8" Oct 30 18:33:53.679656 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.678956 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e91ad8e0-05ce-420f-9594-445f4d48739c-device-dir\") pod \"aws-ebs-csi-driver-node-qtz4g\" (UID: \"e91ad8e0-05ce-420f-9594-445f4d48739c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtz4g" Oct 30 18:33:53.679656 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.678977 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/81f52134-6a09-431f-9f80-412ae7bbead8-host-slash\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:33:53.679656 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.678992 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81f52134-6a09-431f-9f80-412ae7bbead8-host-run-ovn-kubernetes\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:33:53.679656 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.679013 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:33:53.679656 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.679035 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/81f52134-6a09-431f-9f80-412ae7bbead8-host-slash\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:33:53.679656 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.679038 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d2baf406-5501-4717-b5e1-d145edfe9ddc-host-run-netns\") pod \"multus-68r7k\" (UID: \"d2baf406-5501-4717-b5e1-d145edfe9ddc\") " pod="openshift-multus/multus-68r7k" Oct 30 18:33:53.679656 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.679045 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8f76499e-97ad-4c03-8a72-a62f65619596-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:53.679656 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.679066 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/433ed181-b45e-45c0-a593-9cb0b6c09348-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hh862\" (UID: \"433ed181-b45e-45c0-a593-9cb0b6c09348\") " pod="openshift-monitoring/node-exporter-hh862" Oct 30 18:33:53.679656 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.679078 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81f52134-6a09-431f-9f80-412ae7bbead8-host-run-ovn-kubernetes\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:33:53.679656 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.679087 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e91ad8e0-05ce-420f-9594-445f4d48739c-sys-fs\") pod \"aws-ebs-csi-driver-node-qtz4g\" (UID: \"e91ad8e0-05ce-420f-9594-445f4d48739c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtz4g" Oct 30 18:33:53.679656 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.679086 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8f76499e-97ad-4c03-8a72-a62f65619596-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:53.679656 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.679125 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:33:53.679656 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.679128 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-config-volume\") pod \"alertmanager-main-0\" (UID: \"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:33:53.679656 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.679133 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f76499e-97ad-4c03-8a72-a62f65619596-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:53.679656 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.679177 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e91ad8e0-05ce-420f-9594-445f4d48739c-device-dir\") pod \"aws-ebs-csi-driver-node-qtz4g\" (UID: \"e91ad8e0-05ce-420f-9594-445f4d48739c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtz4g" Oct 30 18:33:53.679656 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.679225 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/158becf0-d6cf-4917-8329-4f64ae5a4ca4-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7c6f76d6d-k2h7h\" (UID: \"158becf0-d6cf-4917-8329-4f64ae5a4ca4\") " pod="openshift-monitoring/thanos-querier-7c6f76d6d-k2h7h" Oct 30 18:33:53.679656 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.679241 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d2baf406-5501-4717-b5e1-d145edfe9ddc-host-run-netns\") pod \"multus-68r7k\" (UID: \"d2baf406-5501-4717-b5e1-d145edfe9ddc\") " pod="openshift-multus/multus-68r7k" Oct 30 18:33:53.679656 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.679258 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b826abe-3f79-43ab-8813-2406d5c4dae9-system-cni-dir\") pod \"multus-additional-cni-plugins-nl4m8\" (UID: \"5b826abe-3f79-43ab-8813-2406d5c4dae9\") " pod="openshift-multus/multus-additional-cni-plugins-nl4m8" Oct 30 18:33:53.679656 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.679305 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8f76499e-97ad-4c03-8a72-a62f65619596-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:53.679656 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.679324 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/433ed181-b45e-45c0-a593-9cb0b6c09348-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hh862\" (UID: \"433ed181-b45e-45c0-a593-9cb0b6c09348\") " pod="openshift-monitoring/node-exporter-hh862" Oct 30 18:33:53.679656 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.679386 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/158becf0-d6cf-4917-8329-4f64ae5a4ca4-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7c6f76d6d-k2h7h\" (UID: \"158becf0-d6cf-4917-8329-4f64ae5a4ca4\") " pod="openshift-monitoring/thanos-querier-7c6f76d6d-k2h7h" Oct 30 18:33:53.679656 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.679442 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/158becf0-d6cf-4917-8329-4f64ae5a4ca4-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7c6f76d6d-k2h7h\" (UID: \"158becf0-d6cf-4917-8329-4f64ae5a4ca4\") " pod="openshift-monitoring/thanos-querier-7c6f76d6d-k2h7h" Oct 30 18:33:53.679656 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.679479 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/158becf0-d6cf-4917-8329-4f64ae5a4ca4-tmp\") pod \"thanos-querier-7c6f76d6d-k2h7h\" (UID: \"158becf0-d6cf-4917-8329-4f64ae5a4ca4\") " pod="openshift-monitoring/thanos-querier-7c6f76d6d-k2h7h" Oct 30 18:33:53.679656 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.679504 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f76499e-97ad-4c03-8a72-a62f65619596-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:53.679656 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.679561 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f76499e-97ad-4c03-8a72-a62f65619596-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:53.679656 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.679593 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/158becf0-d6cf-4917-8329-4f64ae5a4ca4-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7c6f76d6d-k2h7h\" (UID: \"158becf0-d6cf-4917-8329-4f64ae5a4ca4\") " pod="openshift-monitoring/thanos-querier-7c6f76d6d-k2h7h" Oct 30 18:33:53.679656 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.679622 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/158becf0-d6cf-4917-8329-4f64ae5a4ca4-tmp\") pod \"thanos-querier-7c6f76d6d-k2h7h\" (UID: \"158becf0-d6cf-4917-8329-4f64ae5a4ca4\") " pod="openshift-monitoring/thanos-querier-7c6f76d6d-k2h7h" Oct 30 18:33:53.679656 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.679642 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8f76499e-97ad-4c03-8a72-a62f65619596-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:53.679656 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.679654 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrzxm\" (UniqueName: \"kubernetes.io/projected/8f76499e-97ad-4c03-8a72-a62f65619596-kube-api-access-jrzxm\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:53.681449 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.679788 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/158becf0-d6cf-4917-8329-4f64ae5a4ca4-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7c6f76d6d-k2h7h\" (UID: \"158becf0-d6cf-4917-8329-4f64ae5a4ca4\") " pod="openshift-monitoring/thanos-querier-7c6f76d6d-k2h7h" Oct 30 18:33:53.681449 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.679818 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d2baf406-5501-4717-b5e1-d145edfe9ddc-etc-kubernetes\") pod \"multus-68r7k\" (UID: \"d2baf406-5501-4717-b5e1-d145edfe9ddc\") " pod="openshift-multus/multus-68r7k" Oct 30 18:33:53.681449 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.679841 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/433ed181-b45e-45c0-a593-9cb0b6c09348-metrics-client-ca\") pod \"node-exporter-hh862\" (UID: \"433ed181-b45e-45c0-a593-9cb0b6c09348\") " pod="openshift-monitoring/node-exporter-hh862" Oct 30 18:33:53.681449 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.679855 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f76499e-97ad-4c03-8a72-a62f65619596-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:53.681449 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.679864 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8f76499e-97ad-4c03-8a72-a62f65619596-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:53.681449 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.679896 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/4f8841c0-4cf3-45d9-9301-957d4fffe1ce-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-7gk7g\" (UID: \"4f8841c0-4cf3-45d9-9301-957d4fffe1ce\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7gk7g" Oct 30 18:33:53.681449 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.679924 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:33:53.681449 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.679952 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6546m\" (UniqueName: \"kubernetes.io/projected/25377c15-6e29-46b3-9f1b-9b1fc38570f5-kube-api-access-6546m\") pod \"monitoring-plugin-77f4f76594-r5d75\" (UID: \"25377c15-6e29-46b3-9f1b-9b1fc38570f5\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-r5d75" Oct 30 18:33:53.681449 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.679956 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/158becf0-d6cf-4917-8329-4f64ae5a4ca4-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7c6f76d6d-k2h7h\" (UID: \"158becf0-d6cf-4917-8329-4f64ae5a4ca4\") " pod="openshift-monitoring/thanos-querier-7c6f76d6d-k2h7h" Oct 30 18:33:53.681449 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.679978 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5b826abe-3f79-43ab-8813-2406d5c4dae9-cnibin\") pod \"multus-additional-cni-plugins-nl4m8\" (UID: \"5b826abe-3f79-43ab-8813-2406d5c4dae9\") " pod="openshift-multus/multus-additional-cni-plugins-nl4m8" Oct 30 18:33:53.681449 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.680033 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8f76499e-97ad-4c03-8a72-a62f65619596-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:53.681449 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.680049 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/433ed181-b45e-45c0-a593-9cb0b6c09348-metrics-client-ca\") pod \"node-exporter-hh862\" (UID: \"433ed181-b45e-45c0-a593-9cb0b6c09348\") " pod="openshift-monitoring/node-exporter-hh862" Oct 30 18:33:53.681449 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.680084 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/eef45986-9475-4e3e-a3f9-fdb682474ff3-etc-sysctl-d\") pod \"tuned-rpzvk\" (UID: \"eef45986-9475-4e3e-a3f9-fdb682474ff3\") " pod="openshift-cluster-node-tuning-operator/tuned-rpzvk" Oct 30 18:33:53.681449 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.680107 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/eef45986-9475-4e3e-a3f9-fdb682474ff3-etc-systemd\") pod \"tuned-rpzvk\" (UID: \"eef45986-9475-4e3e-a3f9-fdb682474ff3\") " pod="openshift-cluster-node-tuning-operator/tuned-rpzvk" Oct 30 18:33:53.681449 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.680115 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/4f8841c0-4cf3-45d9-9301-957d4fffe1ce-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-7gk7g\" (UID: \"4f8841c0-4cf3-45d9-9301-957d4fffe1ce\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7gk7g" Oct 30 18:33:53.681449 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.680128 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d2baf406-5501-4717-b5e1-d145edfe9ddc-host-var-lib-kubelet\") pod \"multus-68r7k\" (UID: \"d2baf406-5501-4717-b5e1-d145edfe9ddc\") " pod="openshift-multus/multus-68r7k" Oct 30 18:33:53.681449 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.680134 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:33:53.681449 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.680199 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d2baf406-5501-4717-b5e1-d145edfe9ddc-etc-kubernetes\") pod \"multus-68r7k\" (UID: \"d2baf406-5501-4717-b5e1-d145edfe9ddc\") " pod="openshift-multus/multus-68r7k" Oct 30 18:33:53.681449 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.680198 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5b826abe-3f79-43ab-8813-2406d5c4dae9-cnibin\") pod \"multus-additional-cni-plugins-nl4m8\" (UID: \"5b826abe-3f79-43ab-8813-2406d5c4dae9\") " pod="openshift-multus/multus-additional-cni-plugins-nl4m8" Oct 30 18:33:53.681449 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.680237 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/eef45986-9475-4e3e-a3f9-fdb682474ff3-etc-systemd\") pod \"tuned-rpzvk\" (UID: \"eef45986-9475-4e3e-a3f9-fdb682474ff3\") " pod="openshift-cluster-node-tuning-operator/tuned-rpzvk" Oct 30 18:33:53.681449 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.680272 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/eef45986-9475-4e3e-a3f9-fdb682474ff3-etc-sysctl-d\") pod \"tuned-rpzvk\" (UID: \"eef45986-9475-4e3e-a3f9-fdb682474ff3\") " pod="openshift-cluster-node-tuning-operator/tuned-rpzvk" Oct 30 18:33:53.681449 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.680276 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/2069a85a-44e7-4c98-9f47-7fad4416c15b-metrics-server-audit-profiles\") pod \"metrics-server-b6d7bcd58-z2xvj\" (UID: \"2069a85a-44e7-4c98-9f47-7fad4416c15b\") " pod="openshift-monitoring/metrics-server-b6d7bcd58-z2xvj" Oct 30 18:33:53.681449 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.680309 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/eef45986-9475-4e3e-a3f9-fdb682474ff3-etc-modprobe-d\") pod \"tuned-rpzvk\" (UID: \"eef45986-9475-4e3e-a3f9-fdb682474ff3\") " pod="openshift-cluster-node-tuning-operator/tuned-rpzvk" Oct 30 18:33:53.681449 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.680337 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d2baf406-5501-4717-b5e1-d145edfe9ddc-multus-socket-dir-parent\") pod \"multus-68r7k\" (UID: \"d2baf406-5501-4717-b5e1-d145edfe9ddc\") " pod="openshift-multus/multus-68r7k" Oct 30 18:33:53.681449 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.680360 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d2baf406-5501-4717-b5e1-d145edfe9ddc-hostroot\") pod \"multus-68r7k\" (UID: \"d2baf406-5501-4717-b5e1-d145edfe9ddc\") " pod="openshift-multus/multus-68r7k" Oct 30 18:33:53.681449 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.680382 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8f76499e-97ad-4c03-8a72-a62f65619596-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:53.681449 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.680390 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d2baf406-5501-4717-b5e1-d145edfe9ddc-host-var-lib-kubelet\") pod \"multus-68r7k\" (UID: \"d2baf406-5501-4717-b5e1-d145edfe9ddc\") " pod="openshift-multus/multus-68r7k" Oct 30 18:33:53.681449 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.680407 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ef86d9fa-a344-4886-b80c-89ce84f418ea-host-slash\") pod \"iptables-alerter-b876b\" (UID: \"ef86d9fa-a344-4886-b80c-89ce84f418ea\") " pod="openshift-network-operator/iptables-alerter-b876b" Oct 30 18:33:53.681449 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.680416 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d2baf406-5501-4717-b5e1-d145edfe9ddc-multus-socket-dir-parent\") pod \"multus-68r7k\" (UID: \"d2baf406-5501-4717-b5e1-d145edfe9ddc\") " pod="openshift-multus/multus-68r7k" Oct 30 18:33:53.681449 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.680429 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d2baf406-5501-4717-b5e1-d145edfe9ddc-hostroot\") pod \"multus-68r7k\" (UID: \"d2baf406-5501-4717-b5e1-d145edfe9ddc\") " pod="openshift-multus/multus-68r7k" Oct 30 18:33:53.681449 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.680430 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/81f52134-6a09-431f-9f80-412ae7bbead8-log-socket\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:33:53.681449 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.680467 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d2baf406-5501-4717-b5e1-d145edfe9ddc-multus-conf-dir\") pod \"multus-68r7k\" (UID: \"d2baf406-5501-4717-b5e1-d145edfe9ddc\") " pod="openshift-multus/multus-68r7k" Oct 30 18:33:53.681449 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.680475 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/eef45986-9475-4e3e-a3f9-fdb682474ff3-etc-modprobe-d\") pod \"tuned-rpzvk\" (UID: \"eef45986-9475-4e3e-a3f9-fdb682474ff3\") " pod="openshift-cluster-node-tuning-operator/tuned-rpzvk" Oct 30 18:33:53.681449 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.680491 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/433ed181-b45e-45c0-a593-9cb0b6c09348-node-exporter-wtmp\") pod \"node-exporter-hh862\" (UID: \"433ed181-b45e-45c0-a593-9cb0b6c09348\") " pod="openshift-monitoring/node-exporter-hh862" Oct 30 18:33:53.681449 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.680480 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/81f52134-6a09-431f-9f80-412ae7bbead8-log-socket\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:33:53.681449 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.680515 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ef86d9fa-a344-4886-b80c-89ce84f418ea-host-slash\") pod \"iptables-alerter-b876b\" (UID: \"ef86d9fa-a344-4886-b80c-89ce84f418ea\") " pod="openshift-network-operator/iptables-alerter-b876b" Oct 30 18:33:53.681449 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.680517 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8f76499e-97ad-4c03-8a72-a62f65619596-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:53.681449 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.680540 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d2baf406-5501-4717-b5e1-d145edfe9ddc-multus-conf-dir\") pod \"multus-68r7k\" (UID: \"d2baf406-5501-4717-b5e1-d145edfe9ddc\") " pod="openshift-multus/multus-68r7k" Oct 30 18:33:53.681449 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.680546 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/2069a85a-44e7-4c98-9f47-7fad4416c15b-metrics-server-audit-profiles\") pod \"metrics-server-b6d7bcd58-z2xvj\" (UID: \"2069a85a-44e7-4c98-9f47-7fad4416c15b\") " pod="openshift-monitoring/metrics-server-b6d7bcd58-z2xvj" Oct 30 18:33:53.681449 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.680546 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/81f52134-6a09-431f-9f80-412ae7bbead8-var-lib-openvswitch\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:33:53.681449 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.680584 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/81f52134-6a09-431f-9f80-412ae7bbead8-var-lib-openvswitch\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:33:53.681449 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.680634 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8f76499e-97ad-4c03-8a72-a62f65619596-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:53.681449 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.680637 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25trq\" (UniqueName: \"kubernetes.io/projected/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-kube-api-access-25trq\") pod \"alertmanager-main-0\" (UID: \"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:33:53.681449 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.680683 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e163025a-b19f-44e2-9058-b65f31bfbcc4-dbus\") pod \"global-pull-secret-syncer-zmdjg\" (UID: \"e163025a-b19f-44e2-9058-b65f31bfbcc4\") " pod="kube-system/global-pull-secret-syncer-zmdjg" Oct 30 18:33:53.681449 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.680701 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8f76499e-97ad-4c03-8a72-a62f65619596-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:53.681449 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.680706 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/81f52134-6a09-431f-9f80-412ae7bbead8-run-ovn\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:33:53.681449 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.680730 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/81f52134-6a09-431f-9f80-412ae7bbead8-node-log\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:33:53.681449 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.680742 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/81f52134-6a09-431f-9f80-412ae7bbead8-run-ovn\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:33:53.681449 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.680749 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d2baf406-5501-4717-b5e1-d145edfe9ddc-host-var-lib-cni-multus\") pod \"multus-68r7k\" (UID: \"d2baf406-5501-4717-b5e1-d145edfe9ddc\") " pod="openshift-multus/multus-68r7k" Oct 30 18:33:53.681449 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.680760 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e163025a-b19f-44e2-9058-b65f31bfbcc4-dbus\") pod \"global-pull-secret-syncer-zmdjg\" (UID: \"e163025a-b19f-44e2-9058-b65f31bfbcc4\") " pod="kube-system/global-pull-secret-syncer-zmdjg" Oct 30 18:33:53.681449 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.680771 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d2baf406-5501-4717-b5e1-d145edfe9ddc-host-var-lib-cni-multus\") pod \"multus-68r7k\" (UID: \"d2baf406-5501-4717-b5e1-d145edfe9ddc\") " pod="openshift-multus/multus-68r7k" Oct 30 18:33:53.681449 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.680776 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4f8841c0-4cf3-45d9-9301-957d4fffe1ce-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-7gk7g\" (UID: \"4f8841c0-4cf3-45d9-9301-957d4fffe1ce\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7gk7g" Oct 30 18:33:53.681449 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.680805 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eef45986-9475-4e3e-a3f9-fdb682474ff3-var-lib-kubelet\") pod \"tuned-rpzvk\" (UID: \"eef45986-9475-4e3e-a3f9-fdb682474ff3\") " pod="openshift-cluster-node-tuning-operator/tuned-rpzvk" Oct 30 18:33:53.681449 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.680807 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/81f52134-6a09-431f-9f80-412ae7bbead8-node-log\") pod \"ovnkube-node-czs4j\" (UID: \"81f52134-6a09-431f-9f80-412ae7bbead8\") " pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:33:53.681449 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.680829 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e91ad8e0-05ce-420f-9594-445f4d48739c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qtz4g\" (UID: \"e91ad8e0-05ce-420f-9594-445f4d48739c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtz4g" Oct 30 18:33:53.681449 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.680873 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d2baf406-5501-4717-b5e1-d145edfe9ddc-host-var-lib-cni-bin\") pod \"multus-68r7k\" (UID: \"d2baf406-5501-4717-b5e1-d145edfe9ddc\") " pod="openshift-multus/multus-68r7k" Oct 30 18:33:53.681449 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.680898 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eef45986-9475-4e3e-a3f9-fdb682474ff3-var-lib-kubelet\") pod \"tuned-rpzvk\" (UID: \"eef45986-9475-4e3e-a3f9-fdb682474ff3\") " pod="openshift-cluster-node-tuning-operator/tuned-rpzvk" Oct 30 18:33:53.681449 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.680898 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g52xm\" (UniqueName: \"kubernetes.io/projected/433ed181-b45e-45c0-a593-9cb0b6c09348-kube-api-access-g52xm\") pod \"node-exporter-hh862\" (UID: \"433ed181-b45e-45c0-a593-9cb0b6c09348\") " pod="openshift-monitoring/node-exporter-hh862" Oct 30 18:33:53.681449 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.680942 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8f76499e-97ad-4c03-8a72-a62f65619596-config\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:53.681449 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.680949 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e91ad8e0-05ce-420f-9594-445f4d48739c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qtz4g\" (UID: \"e91ad8e0-05ce-420f-9594-445f4d48739c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtz4g" Oct 30 18:33:53.681449 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.680955 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d2baf406-5501-4717-b5e1-d145edfe9ddc-host-var-lib-cni-bin\") pod \"multus-68r7k\" (UID: \"d2baf406-5501-4717-b5e1-d145edfe9ddc\") " pod="openshift-multus/multus-68r7k" Oct 30 18:33:53.681449 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.680967 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8f76499e-97ad-4c03-8a72-a62f65619596-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:53.681449 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.680989 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4f8841c0-4cf3-45d9-9301-957d4fffe1ce-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-7gk7g\" (UID: \"4f8841c0-4cf3-45d9-9301-957d4fffe1ce\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7gk7g" Oct 30 18:33:53.681449 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.681060 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2069a85a-44e7-4c98-9f47-7fad4416c15b-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-b6d7bcd58-z2xvj\" (UID: \"2069a85a-44e7-4c98-9f47-7fad4416c15b\") " pod="openshift-monitoring/metrics-server-b6d7bcd58-z2xvj" Oct 30 18:33:53.681449 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.681163 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/8f76499e-97ad-4c03-8a72-a62f65619596-config\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:53.681449 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.681185 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e91ad8e0-05ce-420f-9594-445f4d48739c-etc-selinux\") pod \"aws-ebs-csi-driver-node-qtz4g\" (UID: \"e91ad8e0-05ce-420f-9594-445f4d48739c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtz4g" Oct 30 18:33:53.681449 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.681216 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-web-config\") pod \"alertmanager-main-0\" (UID: \"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:33:53.681449 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.681241 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8f76499e-97ad-4c03-8a72-a62f65619596-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:53.681449 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.681242 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/25377c15-6e29-46b3-9f1b-9b1fc38570f5-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-r5d75\" (UID: \"25377c15-6e29-46b3-9f1b-9b1fc38570f5\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-r5d75" Oct 30 18:33:53.681449 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.681312 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8f76499e-97ad-4c03-8a72-a62f65619596-config-out\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:53.681449 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.681331 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2069a85a-44e7-4c98-9f47-7fad4416c15b-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-b6d7bcd58-z2xvj\" (UID: \"2069a85a-44e7-4c98-9f47-7fad4416c15b\") " pod="openshift-monitoring/metrics-server-b6d7bcd58-z2xvj" Oct 30 18:33:53.681449 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.681285 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e91ad8e0-05ce-420f-9594-445f4d48739c-etc-selinux\") pod \"aws-ebs-csi-driver-node-qtz4g\" (UID: \"e91ad8e0-05ce-420f-9594-445f4d48739c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qtz4g" Oct 30 18:33:53.681449 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.681337 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8f76499e-97ad-4c03-8a72-a62f65619596-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:53.681449 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.681413 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8f76499e-97ad-4c03-8a72-a62f65619596-config-out\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:53.681449 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.681428 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/25377c15-6e29-46b3-9f1b-9b1fc38570f5-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-r5d75\" (UID: \"25377c15-6e29-46b3-9f1b-9b1fc38570f5\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-r5d75" Oct 30 18:33:53.681449 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.681454 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-web-config\") pod \"alertmanager-main-0\" (UID: \"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:33:53.681449 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.681471 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8f76499e-97ad-4c03-8a72-a62f65619596-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:53.683881 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.683766 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8f76499e-97ad-4c03-8a72-a62f65619596-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:53.689101 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.689081 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zndl\" (UniqueName: \"kubernetes.io/projected/158becf0-d6cf-4917-8329-4f64ae5a4ca4-kube-api-access-4zndl\") pod \"thanos-querier-7c6f76d6d-k2h7h\" (UID: \"158becf0-d6cf-4917-8329-4f64ae5a4ca4\") " pod="openshift-monitoring/thanos-querier-7c6f76d6d-k2h7h" Oct 30 18:33:53.689292 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.689272 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wq2pp\" (UniqueName: \"kubernetes.io/projected/2069a85a-44e7-4c98-9f47-7fad4416c15b-kube-api-access-wq2pp\") pod \"metrics-server-b6d7bcd58-z2xvj\" (UID: \"2069a85a-44e7-4c98-9f47-7fad4416c15b\") " pod="openshift-monitoring/metrics-server-b6d7bcd58-z2xvj" Oct 30 18:33:53.689455 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.689441 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9nwr\" (UniqueName: \"kubernetes.io/projected/4f8841c0-4cf3-45d9-9301-957d4fffe1ce-kube-api-access-x9nwr\") pod \"kube-state-metrics-5dc8d7d89b-7gk7g\" (UID: \"4f8841c0-4cf3-45d9-9301-957d4fffe1ce\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7gk7g" Oct 30 18:33:53.689784 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.689766 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrzxm\" (UniqueName: \"kubernetes.io/projected/8f76499e-97ad-4c03-8a72-a62f65619596-kube-api-access-jrzxm\") pod \"prometheus-k8s-0\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:53.690825 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.690805 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25trq\" (UniqueName: \"kubernetes.io/projected/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-kube-api-access-25trq\") pod \"alertmanager-main-0\" (UID: \"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:33:53.694031 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.694007 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g52xm\" (UniqueName: \"kubernetes.io/projected/433ed181-b45e-45c0-a593-9cb0b6c09348-kube-api-access-g52xm\") pod \"node-exporter-hh862\" (UID: \"433ed181-b45e-45c0-a593-9cb0b6c09348\") " pod="openshift-monitoring/node-exporter-hh862" Oct 30 18:33:53.694119 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.694043 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6546m\" (UniqueName: \"kubernetes.io/projected/25377c15-6e29-46b3-9f1b-9b1fc38570f5-kube-api-access-6546m\") pod \"monitoring-plugin-77f4f76594-r5d75\" (UID: \"25377c15-6e29-46b3-9f1b-9b1fc38570f5\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-r5d75" Oct 30 18:33:53.782298 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.782272 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/433ed181-b45e-45c0-a593-9cb0b6c09348-sys\") pod \"node-exporter-hh862\" (UID: \"433ed181-b45e-45c0-a593-9cb0b6c09348\") " pod="openshift-monitoring/node-exporter-hh862" Oct 30 18:33:53.782419 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.782317 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/433ed181-b45e-45c0-a593-9cb0b6c09348-root\") pod \"node-exporter-hh862\" (UID: \"433ed181-b45e-45c0-a593-9cb0b6c09348\") " pod="openshift-monitoring/node-exporter-hh862" Oct 30 18:33:53.782419 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.782354 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/433ed181-b45e-45c0-a593-9cb0b6c09348-node-exporter-wtmp\") pod \"node-exporter-hh862\" (UID: \"433ed181-b45e-45c0-a593-9cb0b6c09348\") " pod="openshift-monitoring/node-exporter-hh862" Oct 30 18:33:53.782419 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.782371 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/433ed181-b45e-45c0-a593-9cb0b6c09348-sys\") pod \"node-exporter-hh862\" (UID: \"433ed181-b45e-45c0-a593-9cb0b6c09348\") " pod="openshift-monitoring/node-exporter-hh862" Oct 30 18:33:53.782508 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.782456 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/433ed181-b45e-45c0-a593-9cb0b6c09348-root\") pod \"node-exporter-hh862\" (UID: \"433ed181-b45e-45c0-a593-9cb0b6c09348\") " pod="openshift-monitoring/node-exporter-hh862" Oct 30 18:33:53.782508 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:53.782487 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/433ed181-b45e-45c0-a593-9cb0b6c09348-node-exporter-wtmp\") pod \"node-exporter-hh862\" (UID: \"433ed181-b45e-45c0-a593-9cb0b6c09348\") " pod="openshift-monitoring/node-exporter-hh862" Oct 30 18:33:53.930089 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:33:53.930056 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 18:33:54.398432 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:54.398401 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7c6f76d6d-k2h7h" Oct 30 18:33:56.144685 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:56.144659 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:33:56.378136 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:56.378104 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-26f57" Oct 30 18:33:56.727404 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:56.727371 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-vfn4g" Oct 30 18:33:57.923074 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:57.923049 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:33:58.190794 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:58.190716 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:33:58.667928 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:58.667892 6960 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 18:33:59.017626 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:59.017562 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-czs4j" Oct 30 18:33:59.739732 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:33:59.739702 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-4g8sn" Oct 30 18:34:01.340302 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:01.340265 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:01.449615 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:01.449581 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:01.474584 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:01.474557 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-r5d75" Oct 30 18:34:06.247668 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:06.247593 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-b6d7bcd58-z2xvj" Oct 30 18:34:10.469555 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:10.469522 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:10.575955 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:10.575925 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:11.540081 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:11.540049 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 18:34:11.540542 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:11.540506 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d" containerName="alertmanager" containerID="cri-o://3fc0ad5914603e602d4c8f2e36d32315d805175b0b3c3bc842e807ce226d5ff7" gracePeriod=120 Oct 30 18:34:11.540638 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:11.540548 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d" containerName="kube-rbac-proxy-web" containerID="cri-o://150cb91b1240c1b880b23a5afdccd8da7736b40f5173f9d83e18b5551baf7d5c" gracePeriod=120 Oct 30 18:34:11.540638 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:11.540541 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d" containerName="config-reloader" containerID="cri-o://28b9e7d1d345ce49931ddbd2d6c992e4a2be94d3a1c3410c8ecea6cfaf4329da" gracePeriod=120 Oct 30 18:34:11.540763 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:11.540544 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d" containerName="kube-rbac-proxy" containerID="cri-o://45f502123b296292940de2da4674a9bd8550f926d07d158169e92083eec6a1d1" gracePeriod=120 Oct 30 18:34:11.540763 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:11.540590 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d" containerName="kube-rbac-proxy-metric" containerID="cri-o://f55ffe8e178a3ea54eee612711a21d86b51a2c7a8aea9865c7b94f0a63bfda88" gracePeriod=120 Oct 30 18:34:11.540763 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:11.540518 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d" containerName="prom-label-proxy" containerID="cri-o://40e4c92d1f7f453ca5dd51f1e5ce8c9880c13a65c1a53e541e169a0f4f73d36e" gracePeriod=120 Oct 30 18:34:11.716811 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:11.716782 6960 generic.go:356] "Generic (PLEG): container finished" podID="7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d" containerID="40e4c92d1f7f453ca5dd51f1e5ce8c9880c13a65c1a53e541e169a0f4f73d36e" exitCode=0 Oct 30 18:34:11.716811 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:11.716806 6960 generic.go:356] "Generic (PLEG): container finished" podID="7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d" containerID="f55ffe8e178a3ea54eee612711a21d86b51a2c7a8aea9865c7b94f0a63bfda88" exitCode=0 Oct 30 18:34:11.716811 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:11.716814 6960 generic.go:356] "Generic (PLEG): container finished" podID="7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d" containerID="45f502123b296292940de2da4674a9bd8550f926d07d158169e92083eec6a1d1" exitCode=0 Oct 30 18:34:11.717022 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:11.716824 6960 generic.go:356] "Generic (PLEG): container finished" podID="7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d" containerID="28b9e7d1d345ce49931ddbd2d6c992e4a2be94d3a1c3410c8ecea6cfaf4329da" exitCode=0 Oct 30 18:34:11.717022 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:11.716831 6960 generic.go:356] "Generic (PLEG): container finished" podID="7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d" containerID="3fc0ad5914603e602d4c8f2e36d32315d805175b0b3c3bc842e807ce226d5ff7" exitCode=0 Oct 30 18:34:11.717022 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:11.716857 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d","Type":"ContainerDied","Data":"40e4c92d1f7f453ca5dd51f1e5ce8c9880c13a65c1a53e541e169a0f4f73d36e"} Oct 30 18:34:11.717022 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:11.716946 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d","Type":"ContainerDied","Data":"f55ffe8e178a3ea54eee612711a21d86b51a2c7a8aea9865c7b94f0a63bfda88"} Oct 30 18:34:11.717022 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:11.716988 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d","Type":"ContainerDied","Data":"45f502123b296292940de2da4674a9bd8550f926d07d158169e92083eec6a1d1"} Oct 30 18:34:11.717201 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:11.717029 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d","Type":"ContainerDied","Data":"28b9e7d1d345ce49931ddbd2d6c992e4a2be94d3a1c3410c8ecea6cfaf4329da"} Oct 30 18:34:11.717201 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:11.717070 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d","Type":"ContainerDied","Data":"3fc0ad5914603e602d4c8f2e36d32315d805175b0b3c3bc842e807ce226d5ff7"} Oct 30 18:34:12.034119 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:12.034085 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:12.174577 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:12.174544 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:12.282352 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:12.282327 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:12.724971 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:12.724935 6960 generic.go:356] "Generic (PLEG): container finished" podID="7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d" containerID="150cb91b1240c1b880b23a5afdccd8da7736b40f5173f9d83e18b5551baf7d5c" exitCode=0 Oct 30 18:34:12.725397 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:12.724975 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d","Type":"ContainerDied","Data":"150cb91b1240c1b880b23a5afdccd8da7736b40f5173f9d83e18b5551baf7d5c"} Oct 30 18:34:12.774507 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:12.774483 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:34:12.823763 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:12.823732 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-config-volume\" (UniqueName: \"kubernetes.io/secret/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-config-volume\") pod \"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d\" (UID: \"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d\") " Oct 30 18:34:12.823957 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:12.823782 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-secret-alertmanager-kube-rbac-proxy\") pod \"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d\" (UID: \"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d\") " Oct 30 18:34:12.823957 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:12.823809 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-secret-alertmanager-main-tls\") pod \"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d\" (UID: \"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d\") " Oct 30 18:34:12.823957 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:12.823856 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-config-out\" (UniqueName: \"kubernetes.io/empty-dir/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-config-out\") pod \"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d\" (UID: \"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d\") " Oct 30 18:34:12.823957 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:12.823934 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d\" (UID: \"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d\") " Oct 30 18:34:12.824177 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:12.823977 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-kube-api-access-25trq\" (UniqueName: \"kubernetes.io/projected/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-kube-api-access-25trq\") pod \"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d\" (UID: \"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d\") " Oct 30 18:34:12.824177 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:12.824020 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-cluster-tls-config\") pod \"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d\" (UID: \"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d\") " Oct 30 18:34:12.824177 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:12.824051 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-secret-alertmanager-kube-rbac-proxy-web\") pod \"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d\" (UID: \"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d\") " Oct 30 18:34:12.824177 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:12.824094 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-alertmanager-trusted-ca-bundle\") pod \"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d\" (UID: \"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d\") " Oct 30 18:34:12.824177 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:12.824137 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-alertmanager-main-db\") pod \"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d\" (UID: \"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d\") " Oct 30 18:34:12.824429 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:12.824201 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-metrics-client-ca\") pod \"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d\" (UID: \"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d\") " Oct 30 18:34:12.824429 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:12.824251 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-tls-assets\" (UniqueName: \"kubernetes.io/projected/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-tls-assets\") pod \"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d\" (UID: \"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d\") " Oct 30 18:34:12.824429 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:12.824288 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-web-config\" (UniqueName: \"kubernetes.io/secret/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-web-config\") pod \"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d\" (UID: \"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d\") " Oct 30 18:34:12.824909 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:12.824873 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-metrics-client-ca" pod "7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d" (UID: "7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:34:12.825446 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:12.825416 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-alertmanager-main-db" pod "7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d" (UID: "7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:34:12.825741 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:12.825714 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-alertmanager-trusted-ca-bundle" pod "7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d" (UID: "7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:34:12.826756 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:12.826727 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-config-volume" pod "7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d" (UID: "7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:34:12.827328 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:12.827295 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-secret-alertmanager-kube-rbac-proxy" pod "7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d" (UID: "7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:34:12.830131 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:12.830105 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-secret-alertmanager-kube-rbac-proxy-metric" pod "7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d" (UID: "7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:34:12.837180 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:12.831450 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-secret-alertmanager-kube-rbac-proxy-web" pod "7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d" (UID: "7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:34:12.837180 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:12.831475 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-kube-api-access-25trq" pod "7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d" (UID: "7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d"). InnerVolumeSpecName "kube-api-access-25trq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:34:12.837180 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:12.831695 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-secret-alertmanager-main-tls" pod "7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d" (UID: "7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:34:12.837180 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:12.831932 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-config-out" pod "7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d" (UID: "7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:34:12.837180 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:12.832075 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-tls-assets" pod "7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d" (UID: "7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:34:12.837180 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:12.836689 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-cluster-tls-config" pod "7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d" (UID: "7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:34:12.842607 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:12.842584 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-web-config" pod "7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d" (UID: "7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:34:12.925176 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:12.925088 6960 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-tls-assets\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:34:12.925176 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:12.925115 6960 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-web-config\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:34:12.925176 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:12.925124 6960 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-config-volume\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:34:12.925176 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:12.925134 6960 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:34:12.925176 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:12.925159 6960 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-secret-alertmanager-main-tls\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:34:12.925176 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:12.925168 6960 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-config-out\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:34:12.925176 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:12.925177 6960 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:34:12.925493 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:12.925187 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25trq\" (UniqueName: \"kubernetes.io/projected/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-kube-api-access-25trq\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:34:12.925493 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:12.925196 6960 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-cluster-tls-config\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:34:12.925493 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:12.925205 6960 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:34:12.925493 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:12.925215 6960 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:34:12.925493 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:12.925224 6960 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-alertmanager-main-db\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:34:12.925493 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:12.925234 6960 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d-metrics-client-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:34:13.731638 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:13.731606 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d","Type":"ContainerDied","Data":"50b0bb5c3b75d6ed563d7acd7bc142c8e6e66f899b26bcdb63b7e097dea2f1fa"} Oct 30 18:34:13.731638 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:13.731638 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:34:13.732048 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:13.731680 6960 scope.go:122] "RemoveContainer" containerID="40e4c92d1f7f453ca5dd51f1e5ce8c9880c13a65c1a53e541e169a0f4f73d36e" Oct 30 18:34:13.740300 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:13.740279 6960 scope.go:122] "RemoveContainer" containerID="f55ffe8e178a3ea54eee612711a21d86b51a2c7a8aea9865c7b94f0a63bfda88" Oct 30 18:34:13.747318 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:13.747303 6960 scope.go:122] "RemoveContainer" containerID="45f502123b296292940de2da4674a9bd8550f926d07d158169e92083eec6a1d1" Oct 30 18:34:13.753786 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:13.753772 6960 scope.go:122] "RemoveContainer" containerID="150cb91b1240c1b880b23a5afdccd8da7736b40f5173f9d83e18b5551baf7d5c" Oct 30 18:34:13.755329 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:13.755307 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 18:34:13.761552 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:13.761492 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 18:34:13.762030 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:13.761986 6960 scope.go:122] "RemoveContainer" containerID="28b9e7d1d345ce49931ddbd2d6c992e4a2be94d3a1c3410c8ecea6cfaf4329da" Oct 30 18:34:13.769067 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:13.769052 6960 scope.go:122] "RemoveContainer" containerID="3fc0ad5914603e602d4c8f2e36d32315d805175b0b3c3bc842e807ce226d5ff7" Oct 30 18:34:13.775799 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:13.775778 6960 scope.go:122] "RemoveContainer" containerID="e01e0e8685fc3c23441371b7c14e6a97b36e570510e3628e07b8c3e5f4bf2e84" Oct 30 18:34:13.791383 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:13.791364 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 18:34:13.791774 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:13.791753 6960 cpu_manager.go:414] "containerMap: removing container" podUID="7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d" containerName="config-reloader" Oct 30 18:34:13.791774 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:13.791774 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d" containerName="config-reloader" Oct 30 18:34:13.791937 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:13.791795 6960 cpu_manager.go:414] "containerMap: removing container" podUID="7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d" containerName="kube-rbac-proxy-web" Oct 30 18:34:13.791937 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:13.791804 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d" containerName="kube-rbac-proxy-web" Oct 30 18:34:13.791937 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:13.791813 6960 cpu_manager.go:414] "containerMap: removing container" podUID="7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d" containerName="init-config-reloader" Oct 30 18:34:13.791937 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:13.791823 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d" containerName="init-config-reloader" Oct 30 18:34:13.791937 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:13.791835 6960 cpu_manager.go:414] "containerMap: removing container" podUID="7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d" containerName="alertmanager" Oct 30 18:34:13.791937 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:13.791841 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d" containerName="alertmanager" Oct 30 18:34:13.791937 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:13.791871 6960 cpu_manager.go:414] "containerMap: removing container" podUID="7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d" containerName="kube-rbac-proxy-metric" Oct 30 18:34:13.791937 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:13.791879 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d" containerName="kube-rbac-proxy-metric" Oct 30 18:34:13.791937 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:13.791906 6960 cpu_manager.go:414] "containerMap: removing container" podUID="7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d" containerName="prom-label-proxy" Oct 30 18:34:13.792301 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:13.791944 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d" containerName="prom-label-proxy" Oct 30 18:34:13.792301 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:13.791971 6960 cpu_manager.go:414] "containerMap: removing container" podUID="7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d" containerName="kube-rbac-proxy" Oct 30 18:34:13.792301 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:13.791979 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d" containerName="kube-rbac-proxy" Oct 30 18:34:13.792301 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:13.792056 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d" containerName="config-reloader" Oct 30 18:34:13.792301 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:13.792068 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d" containerName="kube-rbac-proxy-web" Oct 30 18:34:13.792301 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:13.792077 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d" containerName="init-config-reloader" Oct 30 18:34:13.792301 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:13.792087 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d" containerName="alertmanager" Oct 30 18:34:13.792301 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:13.792098 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d" containerName="kube-rbac-proxy-metric" Oct 30 18:34:13.792301 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:13.792109 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d" containerName="prom-label-proxy" Oct 30 18:34:13.792301 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:13.792118 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d" containerName="kube-rbac-proxy" Oct 30 18:34:13.795746 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:13.795730 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:34:13.800360 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:13.800341 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 18:34:13.800492 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:13.800396 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 18:34:13.800606 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:13.800589 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 18:34:13.801691 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:13.801626 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 18:34:13.801691 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:13.801639 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 18:34:13.801691 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:13.801655 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 18:34:13.801691 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:13.801680 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 18:34:13.801917 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:13.801702 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 18:34:13.801917 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:13.801747 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-h4dl6\"" Oct 30 18:34:13.806583 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:13.806564 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 18:34:13.833617 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:13.833597 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9q4z\" (UniqueName: \"kubernetes.io/projected/13a85849-5c27-4ce8-87ee-059ae2765256-kube-api-access-t9q4z\") pod \"alertmanager-main-0\" (UID: \"13a85849-5c27-4ce8-87ee-059ae2765256\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:34:13.833696 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:13.833626 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/13a85849-5c27-4ce8-87ee-059ae2765256-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"13a85849-5c27-4ce8-87ee-059ae2765256\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:34:13.833738 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:13.833722 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/13a85849-5c27-4ce8-87ee-059ae2765256-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"13a85849-5c27-4ce8-87ee-059ae2765256\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:34:13.833790 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:13.833778 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/13a85849-5c27-4ce8-87ee-059ae2765256-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"13a85849-5c27-4ce8-87ee-059ae2765256\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:34:13.833828 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:13.833799 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/13a85849-5c27-4ce8-87ee-059ae2765256-config-volume\") pod \"alertmanager-main-0\" (UID: \"13a85849-5c27-4ce8-87ee-059ae2765256\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:34:13.833858 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:13.833842 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13a85849-5c27-4ce8-87ee-059ae2765256-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"13a85849-5c27-4ce8-87ee-059ae2765256\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:34:13.833911 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:13.833896 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/13a85849-5c27-4ce8-87ee-059ae2765256-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"13a85849-5c27-4ce8-87ee-059ae2765256\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:34:13.833945 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:13.833928 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/13a85849-5c27-4ce8-87ee-059ae2765256-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"13a85849-5c27-4ce8-87ee-059ae2765256\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:34:13.834013 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:13.833954 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/13a85849-5c27-4ce8-87ee-059ae2765256-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"13a85849-5c27-4ce8-87ee-059ae2765256\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:34:13.834013 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:13.833972 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/13a85849-5c27-4ce8-87ee-059ae2765256-web-config\") pod \"alertmanager-main-0\" (UID: \"13a85849-5c27-4ce8-87ee-059ae2765256\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:34:13.834013 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:13.834000 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/13a85849-5c27-4ce8-87ee-059ae2765256-config-out\") pod \"alertmanager-main-0\" (UID: \"13a85849-5c27-4ce8-87ee-059ae2765256\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:34:13.834159 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:13.834123 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/13a85849-5c27-4ce8-87ee-059ae2765256-tls-assets\") pod \"alertmanager-main-0\" (UID: \"13a85849-5c27-4ce8-87ee-059ae2765256\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:34:13.834202 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:13.834183 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/13a85849-5c27-4ce8-87ee-059ae2765256-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"13a85849-5c27-4ce8-87ee-059ae2765256\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:34:13.854732 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:13.854685 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 18:34:13.934835 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:13.934794 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/13a85849-5c27-4ce8-87ee-059ae2765256-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"13a85849-5c27-4ce8-87ee-059ae2765256\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:34:13.935016 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:13.934874 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/13a85849-5c27-4ce8-87ee-059ae2765256-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"13a85849-5c27-4ce8-87ee-059ae2765256\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:34:13.935016 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:13.934906 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/13a85849-5c27-4ce8-87ee-059ae2765256-config-volume\") pod \"alertmanager-main-0\" (UID: \"13a85849-5c27-4ce8-87ee-059ae2765256\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:34:13.935016 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:13.934931 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13a85849-5c27-4ce8-87ee-059ae2765256-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"13a85849-5c27-4ce8-87ee-059ae2765256\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:34:13.935016 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:13.934959 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/13a85849-5c27-4ce8-87ee-059ae2765256-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"13a85849-5c27-4ce8-87ee-059ae2765256\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:34:13.935016 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:13.934985 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/13a85849-5c27-4ce8-87ee-059ae2765256-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"13a85849-5c27-4ce8-87ee-059ae2765256\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:34:13.935016 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:13.935009 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/13a85849-5c27-4ce8-87ee-059ae2765256-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"13a85849-5c27-4ce8-87ee-059ae2765256\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:34:13.935362 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:13.935035 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/13a85849-5c27-4ce8-87ee-059ae2765256-web-config\") pod \"alertmanager-main-0\" (UID: \"13a85849-5c27-4ce8-87ee-059ae2765256\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:34:13.935362 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:13.935074 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/13a85849-5c27-4ce8-87ee-059ae2765256-config-out\") pod \"alertmanager-main-0\" (UID: \"13a85849-5c27-4ce8-87ee-059ae2765256\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:34:13.935362 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:13.935101 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/13a85849-5c27-4ce8-87ee-059ae2765256-tls-assets\") pod \"alertmanager-main-0\" (UID: \"13a85849-5c27-4ce8-87ee-059ae2765256\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:34:13.935362 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:13.935198 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/13a85849-5c27-4ce8-87ee-059ae2765256-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"13a85849-5c27-4ce8-87ee-059ae2765256\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:34:13.935362 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:13.935230 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9q4z\" (UniqueName: \"kubernetes.io/projected/13a85849-5c27-4ce8-87ee-059ae2765256-kube-api-access-t9q4z\") pod \"alertmanager-main-0\" (UID: \"13a85849-5c27-4ce8-87ee-059ae2765256\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:34:13.935362 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:13.935262 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/13a85849-5c27-4ce8-87ee-059ae2765256-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"13a85849-5c27-4ce8-87ee-059ae2765256\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:34:13.935362 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:13.935321 6960 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 18:34:13.935743 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:13.935419 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/13a85849-5c27-4ce8-87ee-059ae2765256-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"13a85849-5c27-4ce8-87ee-059ae2765256\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:34:13.935743 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:13.935676 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/13a85849-5c27-4ce8-87ee-059ae2765256-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"13a85849-5c27-4ce8-87ee-059ae2765256\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:34:13.936443 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:13.936371 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13a85849-5c27-4ce8-87ee-059ae2765256-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"13a85849-5c27-4ce8-87ee-059ae2765256\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:34:13.938839 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:13.938539 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/13a85849-5c27-4ce8-87ee-059ae2765256-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"13a85849-5c27-4ce8-87ee-059ae2765256\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:34:13.938839 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:13.938659 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/13a85849-5c27-4ce8-87ee-059ae2765256-config-out\") pod \"alertmanager-main-0\" (UID: \"13a85849-5c27-4ce8-87ee-059ae2765256\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:34:13.939002 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:13.938951 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/13a85849-5c27-4ce8-87ee-059ae2765256-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"13a85849-5c27-4ce8-87ee-059ae2765256\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:34:13.939063 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:13.939016 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/13a85849-5c27-4ce8-87ee-059ae2765256-config-volume\") pod \"alertmanager-main-0\" (UID: \"13a85849-5c27-4ce8-87ee-059ae2765256\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:34:13.939063 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:13.939026 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/13a85849-5c27-4ce8-87ee-059ae2765256-web-config\") pod \"alertmanager-main-0\" (UID: \"13a85849-5c27-4ce8-87ee-059ae2765256\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:34:13.939063 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:13.939052 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/13a85849-5c27-4ce8-87ee-059ae2765256-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"13a85849-5c27-4ce8-87ee-059ae2765256\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:34:13.939274 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:13.939258 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/13a85849-5c27-4ce8-87ee-059ae2765256-tls-assets\") pod \"alertmanager-main-0\" (UID: \"13a85849-5c27-4ce8-87ee-059ae2765256\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:34:13.939274 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:13.939261 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/13a85849-5c27-4ce8-87ee-059ae2765256-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"13a85849-5c27-4ce8-87ee-059ae2765256\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:34:13.940197 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:13.940181 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/13a85849-5c27-4ce8-87ee-059ae2765256-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"13a85849-5c27-4ce8-87ee-059ae2765256\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:34:13.947687 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:13.947662 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9q4z\" (UniqueName: \"kubernetes.io/projected/13a85849-5c27-4ce8-87ee-059ae2765256-kube-api-access-t9q4z\") pod \"alertmanager-main-0\" (UID: \"13a85849-5c27-4ce8-87ee-059ae2765256\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:34:14.120359 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:14.120283 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:34:14.403564 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:14.403499 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 18:34:14.407012 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:34:14.406985 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13a85849_5c27_4ce8_87ee_059ae2765256.slice/crio-120c2bdccae078d07c49b3379da22753797e3b579fa2e79172513b4a4350356e WatchSource:0}: Error finding container 120c2bdccae078d07c49b3379da22753797e3b579fa2e79172513b4a4350356e: Status 404 returned error can't find the container with id 120c2bdccae078d07c49b3379da22753797e3b579fa2e79172513b4a4350356e Oct 30 18:34:14.589382 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:14.589355 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d" path="/var/lib/kubelet/pods/7b08b5b7-a078-47c9-8a1f-4cc73c36fb7d/volumes" Oct 30 18:34:14.736282 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:14.736211 6960 generic.go:356] "Generic (PLEG): container finished" podID="13a85849-5c27-4ce8-87ee-059ae2765256" containerID="ae52f7f1664d1f9ccf6eeb767a5cf24ab8d0f968fd04916c13caa9a519fea982" exitCode=0 Oct 30 18:34:14.736282 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:14.736262 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"13a85849-5c27-4ce8-87ee-059ae2765256","Type":"ContainerDied","Data":"ae52f7f1664d1f9ccf6eeb767a5cf24ab8d0f968fd04916c13caa9a519fea982"} Oct 30 18:34:14.736766 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:14.736346 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"13a85849-5c27-4ce8-87ee-059ae2765256","Type":"ContainerStarted","Data":"120c2bdccae078d07c49b3379da22753797e3b579fa2e79172513b4a4350356e"} Oct 30 18:34:15.561456 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:15.561417 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-85d6484797-5gnbj"] Oct 30 18:34:15.565752 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:15.565730 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-85d6484797-5gnbj" Oct 30 18:34:15.571075 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:15.571042 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Oct 30 18:34:15.572880 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:15.572859 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Oct 30 18:34:15.581275 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:15.581244 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-85d6484797-5gnbj"] Oct 30 18:34:15.649388 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:15.649366 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/48000d54-fb98-4094-8160-b44be17f190c-telemeter-client-tls\") pod \"telemeter-client-85d6484797-5gnbj\" (UID: \"48000d54-fb98-4094-8160-b44be17f190c\") " pod="openshift-monitoring/telemeter-client-85d6484797-5gnbj" Oct 30 18:34:15.649499 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:15.649395 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/48000d54-fb98-4094-8160-b44be17f190c-federate-client-tls\") pod \"telemeter-client-85d6484797-5gnbj\" (UID: \"48000d54-fb98-4094-8160-b44be17f190c\") " pod="openshift-monitoring/telemeter-client-85d6484797-5gnbj" Oct 30 18:34:15.649499 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:15.649414 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48000d54-fb98-4094-8160-b44be17f190c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-85d6484797-5gnbj\" (UID: \"48000d54-fb98-4094-8160-b44be17f190c\") " pod="openshift-monitoring/telemeter-client-85d6484797-5gnbj" Oct 30 18:34:15.649596 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:15.649508 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mf78x\" (UniqueName: \"kubernetes.io/projected/48000d54-fb98-4094-8160-b44be17f190c-kube-api-access-mf78x\") pod \"telemeter-client-85d6484797-5gnbj\" (UID: \"48000d54-fb98-4094-8160-b44be17f190c\") " pod="openshift-monitoring/telemeter-client-85d6484797-5gnbj" Oct 30 18:34:15.649596 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:15.649538 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/48000d54-fb98-4094-8160-b44be17f190c-secret-telemeter-client\") pod \"telemeter-client-85d6484797-5gnbj\" (UID: \"48000d54-fb98-4094-8160-b44be17f190c\") " pod="openshift-monitoring/telemeter-client-85d6484797-5gnbj" Oct 30 18:34:15.649685 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:15.649599 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/48000d54-fb98-4094-8160-b44be17f190c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-85d6484797-5gnbj\" (UID: \"48000d54-fb98-4094-8160-b44be17f190c\") " pod="openshift-monitoring/telemeter-client-85d6484797-5gnbj" Oct 30 18:34:15.649685 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:15.649648 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/48000d54-fb98-4094-8160-b44be17f190c-metrics-client-ca\") pod \"telemeter-client-85d6484797-5gnbj\" (UID: \"48000d54-fb98-4094-8160-b44be17f190c\") " pod="openshift-monitoring/telemeter-client-85d6484797-5gnbj" Oct 30 18:34:15.649789 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:15.649727 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48000d54-fb98-4094-8160-b44be17f190c-serving-certs-ca-bundle\") pod \"telemeter-client-85d6484797-5gnbj\" (UID: \"48000d54-fb98-4094-8160-b44be17f190c\") " pod="openshift-monitoring/telemeter-client-85d6484797-5gnbj" Oct 30 18:34:15.654099 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:15.654080 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Oct 30 18:34:15.654200 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:15.654083 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Oct 30 18:34:15.654200 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:15.654084 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Oct 30 18:34:15.654293 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:15.654208 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Oct 30 18:34:15.654293 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:15.654251 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-g625p\"" Oct 30 18:34:15.747633 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:15.747607 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"13a85849-5c27-4ce8-87ee-059ae2765256","Type":"ContainerStarted","Data":"ba27b685105275cdd9377ed4024716f766d455dfa2ea5dd4fb45c4478755654f"} Oct 30 18:34:15.747930 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:15.747683 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"13a85849-5c27-4ce8-87ee-059ae2765256","Type":"ContainerStarted","Data":"a694f78d16c74056f27d4044d2feefa9d6381d9bf834882d5557dc77764ed47e"} Oct 30 18:34:15.747930 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:15.747711 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"13a85849-5c27-4ce8-87ee-059ae2765256","Type":"ContainerStarted","Data":"404503bdd47607a634686ae6416e9aaa32d4560c44ecce6c5aec525987f43ea4"} Oct 30 18:34:15.747930 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:15.747735 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"13a85849-5c27-4ce8-87ee-059ae2765256","Type":"ContainerStarted","Data":"18d4843972c91af0c62282e6fc8ed0e51d24c73284446a711f0f8924ede31ac3"} Oct 30 18:34:15.747930 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:15.747759 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"13a85849-5c27-4ce8-87ee-059ae2765256","Type":"ContainerStarted","Data":"666427021d8653449b040f06d822a4df6dfe9ac5df9a9c6e715560e6b99ea7f5"} Oct 30 18:34:15.747930 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:15.747780 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"13a85849-5c27-4ce8-87ee-059ae2765256","Type":"ContainerStarted","Data":"2fe9bab1e7bdf0eeaf29adec846d3ec2696f34682a3d2be730b8ae8fe15930f5"} Oct 30 18:34:15.750211 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:15.750193 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/48000d54-fb98-4094-8160-b44be17f190c-telemeter-client-tls\") pod \"telemeter-client-85d6484797-5gnbj\" (UID: \"48000d54-fb98-4094-8160-b44be17f190c\") " pod="openshift-monitoring/telemeter-client-85d6484797-5gnbj" Oct 30 18:34:15.750259 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:15.750218 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/48000d54-fb98-4094-8160-b44be17f190c-federate-client-tls\") pod \"telemeter-client-85d6484797-5gnbj\" (UID: \"48000d54-fb98-4094-8160-b44be17f190c\") " pod="openshift-monitoring/telemeter-client-85d6484797-5gnbj" Oct 30 18:34:15.750259 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:15.750234 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48000d54-fb98-4094-8160-b44be17f190c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-85d6484797-5gnbj\" (UID: \"48000d54-fb98-4094-8160-b44be17f190c\") " pod="openshift-monitoring/telemeter-client-85d6484797-5gnbj" Oct 30 18:34:15.750353 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:15.750273 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mf78x\" (UniqueName: \"kubernetes.io/projected/48000d54-fb98-4094-8160-b44be17f190c-kube-api-access-mf78x\") pod \"telemeter-client-85d6484797-5gnbj\" (UID: \"48000d54-fb98-4094-8160-b44be17f190c\") " pod="openshift-monitoring/telemeter-client-85d6484797-5gnbj" Oct 30 18:34:15.750353 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:15.750303 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/48000d54-fb98-4094-8160-b44be17f190c-secret-telemeter-client\") pod \"telemeter-client-85d6484797-5gnbj\" (UID: \"48000d54-fb98-4094-8160-b44be17f190c\") " pod="openshift-monitoring/telemeter-client-85d6484797-5gnbj" Oct 30 18:34:15.750353 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:15.750327 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/48000d54-fb98-4094-8160-b44be17f190c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-85d6484797-5gnbj\" (UID: \"48000d54-fb98-4094-8160-b44be17f190c\") " pod="openshift-monitoring/telemeter-client-85d6484797-5gnbj" Oct 30 18:34:15.750488 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:15.750354 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/48000d54-fb98-4094-8160-b44be17f190c-metrics-client-ca\") pod \"telemeter-client-85d6484797-5gnbj\" (UID: \"48000d54-fb98-4094-8160-b44be17f190c\") " pod="openshift-monitoring/telemeter-client-85d6484797-5gnbj" Oct 30 18:34:15.750488 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:15.750386 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48000d54-fb98-4094-8160-b44be17f190c-serving-certs-ca-bundle\") pod \"telemeter-client-85d6484797-5gnbj\" (UID: \"48000d54-fb98-4094-8160-b44be17f190c\") " pod="openshift-monitoring/telemeter-client-85d6484797-5gnbj" Oct 30 18:34:15.751154 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:15.751099 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48000d54-fb98-4094-8160-b44be17f190c-serving-certs-ca-bundle\") pod \"telemeter-client-85d6484797-5gnbj\" (UID: \"48000d54-fb98-4094-8160-b44be17f190c\") " pod="openshift-monitoring/telemeter-client-85d6484797-5gnbj" Oct 30 18:34:15.751387 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:15.751365 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/48000d54-fb98-4094-8160-b44be17f190c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-85d6484797-5gnbj\" (UID: \"48000d54-fb98-4094-8160-b44be17f190c\") " pod="openshift-monitoring/telemeter-client-85d6484797-5gnbj" Oct 30 18:34:15.752799 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:15.752772 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/48000d54-fb98-4094-8160-b44be17f190c-metrics-client-ca\") pod \"telemeter-client-85d6484797-5gnbj\" (UID: \"48000d54-fb98-4094-8160-b44be17f190c\") " pod="openshift-monitoring/telemeter-client-85d6484797-5gnbj" Oct 30 18:34:15.752914 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:15.752891 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/48000d54-fb98-4094-8160-b44be17f190c-secret-telemeter-client\") pod \"telemeter-client-85d6484797-5gnbj\" (UID: \"48000d54-fb98-4094-8160-b44be17f190c\") " pod="openshift-monitoring/telemeter-client-85d6484797-5gnbj" Oct 30 18:34:15.753050 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:15.753029 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/48000d54-fb98-4094-8160-b44be17f190c-federate-client-tls\") pod \"telemeter-client-85d6484797-5gnbj\" (UID: \"48000d54-fb98-4094-8160-b44be17f190c\") " pod="openshift-monitoring/telemeter-client-85d6484797-5gnbj" Oct 30 18:34:15.753396 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:15.753376 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/48000d54-fb98-4094-8160-b44be17f190c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-85d6484797-5gnbj\" (UID: \"48000d54-fb98-4094-8160-b44be17f190c\") " pod="openshift-monitoring/telemeter-client-85d6484797-5gnbj" Oct 30 18:34:15.753445 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:15.753380 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/48000d54-fb98-4094-8160-b44be17f190c-telemeter-client-tls\") pod \"telemeter-client-85d6484797-5gnbj\" (UID: \"48000d54-fb98-4094-8160-b44be17f190c\") " pod="openshift-monitoring/telemeter-client-85d6484797-5gnbj" Oct 30 18:34:15.762004 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:15.761981 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mf78x\" (UniqueName: \"kubernetes.io/projected/48000d54-fb98-4094-8160-b44be17f190c-kube-api-access-mf78x\") pod \"telemeter-client-85d6484797-5gnbj\" (UID: \"48000d54-fb98-4094-8160-b44be17f190c\") " pod="openshift-monitoring/telemeter-client-85d6484797-5gnbj" Oct 30 18:34:15.780698 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:15.780404 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.767223476 podStartE2EDuration="2.780388936s" totalImagesPullingTime="0s" totalInitContainerRuntime="13.16546ms" isStatefulPod=false podCreationTimestamp="2026-10-30 18:34:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:34:15.779513069 +0000 UTC m=+23.308544210" watchObservedRunningTime="2026-10-30 18:34:15.780388936 +0000 UTC m=+23.309420069" Oct 30 18:34:15.881849 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:15.881782 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-85d6484797-5gnbj" Oct 30 18:34:16.024322 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.024293 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 18:34:16.024608 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.024577 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8f76499e-97ad-4c03-8a72-a62f65619596" containerName="prometheus" containerID="cri-o://c5aea0259e406d5b190a03fce661f9a4052cc4d10160e32c100ffdf12f7433d5" gracePeriod=600 Oct 30 18:34:16.024608 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.024595 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8f76499e-97ad-4c03-8a72-a62f65619596" containerName="kube-rbac-proxy" containerID="cri-o://be397bdaf885f8c055c80395dea268c2f30e89f2cb1526720d2440af75217759" gracePeriod=600 Oct 30 18:34:16.024802 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.024601 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8f76499e-97ad-4c03-8a72-a62f65619596" containerName="thanos-sidecar" containerID="cri-o://a7ec88732cafa80ab354e8864f75c1833177871423f5ca944fbf24dc7bb44a2b" gracePeriod=600 Oct 30 18:34:16.024802 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.024619 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8f76499e-97ad-4c03-8a72-a62f65619596" containerName="config-reloader" containerID="cri-o://b049f7e3597139cac61bd3d2c74278bbfd9a2df99c6ffeaef14d0bda8e236eef" gracePeriod=600 Oct 30 18:34:16.024802 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.024626 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8f76499e-97ad-4c03-8a72-a62f65619596" containerName="kube-rbac-proxy-web" containerID="cri-o://2b7e0cbea7cc9adfbc4ddedcd2bb9ab689149d004cf05ae4c38d9a2c2ec563f5" gracePeriod=600 Oct 30 18:34:16.024802 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.024710 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8f76499e-97ad-4c03-8a72-a62f65619596" containerName="kube-rbac-proxy-thanos" containerID="cri-o://563244759dbdbc2c58d0b4ba9e94b736d2bb8e69feb6e845291e2e2d8bf3fcb2" gracePeriod=600 Oct 30 18:34:16.125984 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:34:16.125936 6960 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="c5aea0259e406d5b190a03fce661f9a4052cc4d10160e32c100ffdf12f7433d5" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"] Oct 30 18:34:16.126074 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:34:16.126002 6960 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="c5aea0259e406d5b190a03fce661f9a4052cc4d10160e32c100ffdf12f7433d5" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"] Oct 30 18:34:16.126074 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:34:16.126019 6960 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="c5aea0259e406d5b190a03fce661f9a4052cc4d10160e32c100ffdf12f7433d5" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"] Oct 30 18:34:16.126074 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.126033 6960 prober.go:106] "Probe errored" probeType="Readiness" pod="openshift-monitoring/prometheus-k8s-0" podUID="8f76499e-97ad-4c03-8a72-a62f65619596" containerName="prometheus" probeResult="unknown" err="rpc error: code = Canceled desc = context canceled" Oct 30 18:34:16.157691 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.157638 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-85d6484797-5gnbj"] Oct 30 18:34:16.164942 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.164920 6960 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 18:34:16.269234 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.269210 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:16.356160 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.356119 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f76499e-97ad-4c03-8a72-a62f65619596-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8f76499e-97ad-4c03-8a72-a62f65619596-prometheus-k8s-rulefiles-2\") pod \"8f76499e-97ad-4c03-8a72-a62f65619596\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " Oct 30 18:34:16.356317 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.356182 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f76499e-97ad-4c03-8a72-a62f65619596-tls-assets\" (UniqueName: \"kubernetes.io/projected/8f76499e-97ad-4c03-8a72-a62f65619596-tls-assets\") pod \"8f76499e-97ad-4c03-8a72-a62f65619596\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " Oct 30 18:34:16.356317 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.356281 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f76499e-97ad-4c03-8a72-a62f65619596-kube-api-access-jrzxm\" (UniqueName: \"kubernetes.io/projected/8f76499e-97ad-4c03-8a72-a62f65619596-kube-api-access-jrzxm\") pod \"8f76499e-97ad-4c03-8a72-a62f65619596\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " Oct 30 18:34:16.356317 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.356313 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f76499e-97ad-4c03-8a72-a62f65619596-config\" (UniqueName: \"kubernetes.io/secret/8f76499e-97ad-4c03-8a72-a62f65619596-config\") pod \"8f76499e-97ad-4c03-8a72-a62f65619596\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " Oct 30 18:34:16.356474 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.356343 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f76499e-97ad-4c03-8a72-a62f65619596-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8f76499e-97ad-4c03-8a72-a62f65619596-secret-kube-rbac-proxy\") pod \"8f76499e-97ad-4c03-8a72-a62f65619596\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " Oct 30 18:34:16.356530 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.356476 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f76499e-97ad-4c03-8a72-a62f65619596-config-out\" (UniqueName: \"kubernetes.io/empty-dir/8f76499e-97ad-4c03-8a72-a62f65619596-config-out\") pod \"8f76499e-97ad-4c03-8a72-a62f65619596\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " Oct 30 18:34:16.356530 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.356506 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f76499e-97ad-4c03-8a72-a62f65619596-prometheus-k8s-rulefiles-2" pod "8f76499e-97ad-4c03-8a72-a62f65619596" (UID: "8f76499e-97ad-4c03-8a72-a62f65619596"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:34:16.356530 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.356524 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f76499e-97ad-4c03-8a72-a62f65619596-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8f76499e-97ad-4c03-8a72-a62f65619596-prometheus-k8s-rulefiles-1\") pod \"8f76499e-97ad-4c03-8a72-a62f65619596\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " Oct 30 18:34:16.356687 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.356569 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f76499e-97ad-4c03-8a72-a62f65619596-web-config\" (UniqueName: \"kubernetes.io/secret/8f76499e-97ad-4c03-8a72-a62f65619596-web-config\") pod \"8f76499e-97ad-4c03-8a72-a62f65619596\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " Oct 30 18:34:16.356687 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.356596 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f76499e-97ad-4c03-8a72-a62f65619596-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f76499e-97ad-4c03-8a72-a62f65619596-prometheus-trusted-ca-bundle\") pod \"8f76499e-97ad-4c03-8a72-a62f65619596\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " Oct 30 18:34:16.356967 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.356904 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f76499e-97ad-4c03-8a72-a62f65619596-prometheus-k8s-rulefiles-1" pod "8f76499e-97ad-4c03-8a72-a62f65619596" (UID: "8f76499e-97ad-4c03-8a72-a62f65619596"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:34:16.356967 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.356951 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f76499e-97ad-4c03-8a72-a62f65619596-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8f76499e-97ad-4c03-8a72-a62f65619596-secret-grpc-tls\") pod \"8f76499e-97ad-4c03-8a72-a62f65619596\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " Oct 30 18:34:16.356967 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.356953 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f76499e-97ad-4c03-8a72-a62f65619596-prometheus-trusted-ca-bundle" pod "8f76499e-97ad-4c03-8a72-a62f65619596" (UID: "8f76499e-97ad-4c03-8a72-a62f65619596"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:34:16.357117 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.356987 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f76499e-97ad-4c03-8a72-a62f65619596-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8f76499e-97ad-4c03-8a72-a62f65619596-configmap-metrics-client-ca\") pod \"8f76499e-97ad-4c03-8a72-a62f65619596\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " Oct 30 18:34:16.357117 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.357034 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f76499e-97ad-4c03-8a72-a62f65619596-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8f76499e-97ad-4c03-8a72-a62f65619596-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"8f76499e-97ad-4c03-8a72-a62f65619596\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " Oct 30 18:34:16.357117 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.357081 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f76499e-97ad-4c03-8a72-a62f65619596-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8f76499e-97ad-4c03-8a72-a62f65619596-secret-prometheus-k8s-tls\") pod \"8f76499e-97ad-4c03-8a72-a62f65619596\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " Oct 30 18:34:16.357292 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.357131 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f76499e-97ad-4c03-8a72-a62f65619596-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f76499e-97ad-4c03-8a72-a62f65619596-configmap-serving-certs-ca-bundle\") pod \"8f76499e-97ad-4c03-8a72-a62f65619596\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " Oct 30 18:34:16.357292 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.357188 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f76499e-97ad-4c03-8a72-a62f65619596-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8f76499e-97ad-4c03-8a72-a62f65619596-prometheus-k8s-db\") pod \"8f76499e-97ad-4c03-8a72-a62f65619596\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " Oct 30 18:34:16.357292 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.357218 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f76499e-97ad-4c03-8a72-a62f65619596-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8f76499e-97ad-4c03-8a72-a62f65619596-prometheus-k8s-rulefiles-0\") pod \"8f76499e-97ad-4c03-8a72-a62f65619596\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " Oct 30 18:34:16.357292 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.357248 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f76499e-97ad-4c03-8a72-a62f65619596-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8f76499e-97ad-4c03-8a72-a62f65619596-secret-metrics-client-certs\") pod \"8f76499e-97ad-4c03-8a72-a62f65619596\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " Oct 30 18:34:16.357292 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.357277 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f76499e-97ad-4c03-8a72-a62f65619596-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8f76499e-97ad-4c03-8a72-a62f65619596-thanos-prometheus-http-client-file\") pod \"8f76499e-97ad-4c03-8a72-a62f65619596\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " Oct 30 18:34:16.357540 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.357377 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f76499e-97ad-4c03-8a72-a62f65619596-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f76499e-97ad-4c03-8a72-a62f65619596-configmap-kubelet-serving-ca-bundle\") pod \"8f76499e-97ad-4c03-8a72-a62f65619596\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " Oct 30 18:34:16.357540 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.357410 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f76499e-97ad-4c03-8a72-a62f65619596-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8f76499e-97ad-4c03-8a72-a62f65619596-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"8f76499e-97ad-4c03-8a72-a62f65619596\" (UID: \"8f76499e-97ad-4c03-8a72-a62f65619596\") " Oct 30 18:34:16.357540 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.357456 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f76499e-97ad-4c03-8a72-a62f65619596-configmap-metrics-client-ca" pod "8f76499e-97ad-4c03-8a72-a62f65619596" (UID: "8f76499e-97ad-4c03-8a72-a62f65619596"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:34:16.359852 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.357852 6960 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8f76499e-97ad-4c03-8a72-a62f65619596-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:34:16.359852 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.357879 6960 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f76499e-97ad-4c03-8a72-a62f65619596-prometheus-trusted-ca-bundle\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:34:16.359852 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.357896 6960 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8f76499e-97ad-4c03-8a72-a62f65619596-configmap-metrics-client-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:34:16.359852 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.358324 6960 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8f76499e-97ad-4c03-8a72-a62f65619596-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:34:16.359852 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.358900 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f76499e-97ad-4c03-8a72-a62f65619596-prometheus-k8s-db" pod "8f76499e-97ad-4c03-8a72-a62f65619596" (UID: "8f76499e-97ad-4c03-8a72-a62f65619596"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:34:16.359852 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.359742 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f76499e-97ad-4c03-8a72-a62f65619596-configmap-serving-certs-ca-bundle" pod "8f76499e-97ad-4c03-8a72-a62f65619596" (UID: "8f76499e-97ad-4c03-8a72-a62f65619596"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:34:16.360542 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.360044 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f76499e-97ad-4c03-8a72-a62f65619596-config" pod "8f76499e-97ad-4c03-8a72-a62f65619596" (UID: "8f76499e-97ad-4c03-8a72-a62f65619596"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:34:16.360542 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.360332 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f76499e-97ad-4c03-8a72-a62f65619596-configmap-kubelet-serving-ca-bundle" pod "8f76499e-97ad-4c03-8a72-a62f65619596" (UID: "8f76499e-97ad-4c03-8a72-a62f65619596"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:34:16.360837 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.360781 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f76499e-97ad-4c03-8a72-a62f65619596-prometheus-k8s-rulefiles-0" pod "8f76499e-97ad-4c03-8a72-a62f65619596" (UID: "8f76499e-97ad-4c03-8a72-a62f65619596"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:34:16.361098 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.361061 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f76499e-97ad-4c03-8a72-a62f65619596-config-out" pod "8f76499e-97ad-4c03-8a72-a62f65619596" (UID: "8f76499e-97ad-4c03-8a72-a62f65619596"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:34:16.361293 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.361266 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f76499e-97ad-4c03-8a72-a62f65619596-tls-assets" pod "8f76499e-97ad-4c03-8a72-a62f65619596" (UID: "8f76499e-97ad-4c03-8a72-a62f65619596"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:34:16.361507 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.361487 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f76499e-97ad-4c03-8a72-a62f65619596-kube-api-access-jrzxm" pod "8f76499e-97ad-4c03-8a72-a62f65619596" (UID: "8f76499e-97ad-4c03-8a72-a62f65619596"). InnerVolumeSpecName "kube-api-access-jrzxm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:34:16.361742 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.361699 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f76499e-97ad-4c03-8a72-a62f65619596-secret-prometheus-k8s-thanos-sidecar-tls" pod "8f76499e-97ad-4c03-8a72-a62f65619596" (UID: "8f76499e-97ad-4c03-8a72-a62f65619596"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:34:16.361832 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.361816 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f76499e-97ad-4c03-8a72-a62f65619596-secret-prometheus-k8s-kube-rbac-proxy-web" pod "8f76499e-97ad-4c03-8a72-a62f65619596" (UID: "8f76499e-97ad-4c03-8a72-a62f65619596"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:34:16.361997 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.361971 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f76499e-97ad-4c03-8a72-a62f65619596-secret-grpc-tls" pod "8f76499e-97ad-4c03-8a72-a62f65619596" (UID: "8f76499e-97ad-4c03-8a72-a62f65619596"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:34:16.362107 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.362043 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f76499e-97ad-4c03-8a72-a62f65619596-secret-kube-rbac-proxy" pod "8f76499e-97ad-4c03-8a72-a62f65619596" (UID: "8f76499e-97ad-4c03-8a72-a62f65619596"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:34:16.362638 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.362613 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f76499e-97ad-4c03-8a72-a62f65619596-thanos-prometheus-http-client-file" pod "8f76499e-97ad-4c03-8a72-a62f65619596" (UID: "8f76499e-97ad-4c03-8a72-a62f65619596"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:34:16.362791 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.362775 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f76499e-97ad-4c03-8a72-a62f65619596-secret-metrics-client-certs" pod "8f76499e-97ad-4c03-8a72-a62f65619596" (UID: "8f76499e-97ad-4c03-8a72-a62f65619596"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:34:16.362917 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.362890 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f76499e-97ad-4c03-8a72-a62f65619596-secret-prometheus-k8s-tls" pod "8f76499e-97ad-4c03-8a72-a62f65619596" (UID: "8f76499e-97ad-4c03-8a72-a62f65619596"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:34:16.372935 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.372911 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f76499e-97ad-4c03-8a72-a62f65619596-web-config" pod "8f76499e-97ad-4c03-8a72-a62f65619596" (UID: "8f76499e-97ad-4c03-8a72-a62f65619596"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:34:16.459270 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.459217 6960 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8f76499e-97ad-4c03-8a72-a62f65619596-secret-prometheus-k8s-tls\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:34:16.459270 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.459237 6960 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f76499e-97ad-4c03-8a72-a62f65619596-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:34:16.459270 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.459248 6960 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8f76499e-97ad-4c03-8a72-a62f65619596-prometheus-k8s-db\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:34:16.459270 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.459257 6960 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8f76499e-97ad-4c03-8a72-a62f65619596-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:34:16.459270 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.459267 6960 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8f76499e-97ad-4c03-8a72-a62f65619596-secret-metrics-client-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:34:16.459468 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.459276 6960 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8f76499e-97ad-4c03-8a72-a62f65619596-thanos-prometheus-http-client-file\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:34:16.459468 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.459286 6960 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f76499e-97ad-4c03-8a72-a62f65619596-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:34:16.459468 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.459295 6960 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8f76499e-97ad-4c03-8a72-a62f65619596-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:34:16.459468 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.459304 6960 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8f76499e-97ad-4c03-8a72-a62f65619596-tls-assets\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:34:16.459468 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.459312 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrzxm\" (UniqueName: \"kubernetes.io/projected/8f76499e-97ad-4c03-8a72-a62f65619596-kube-api-access-jrzxm\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:34:16.459468 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.459321 6960 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/8f76499e-97ad-4c03-8a72-a62f65619596-config\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:34:16.459468 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.459329 6960 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8f76499e-97ad-4c03-8a72-a62f65619596-secret-kube-rbac-proxy\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:34:16.459468 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.459337 6960 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8f76499e-97ad-4c03-8a72-a62f65619596-config-out\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:34:16.459468 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.459346 6960 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8f76499e-97ad-4c03-8a72-a62f65619596-web-config\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:34:16.459468 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.459354 6960 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8f76499e-97ad-4c03-8a72-a62f65619596-secret-grpc-tls\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:34:16.459468 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.459362 6960 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8f76499e-97ad-4c03-8a72-a62f65619596-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:34:16.756756 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.756675 6960 generic.go:356] "Generic (PLEG): container finished" podID="8f76499e-97ad-4c03-8a72-a62f65619596" containerID="563244759dbdbc2c58d0b4ba9e94b736d2bb8e69feb6e845291e2e2d8bf3fcb2" exitCode=0 Oct 30 18:34:16.756756 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.756707 6960 generic.go:356] "Generic (PLEG): container finished" podID="8f76499e-97ad-4c03-8a72-a62f65619596" containerID="be397bdaf885f8c055c80395dea268c2f30e89f2cb1526720d2440af75217759" exitCode=0 Oct 30 18:34:16.756756 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.756720 6960 generic.go:356] "Generic (PLEG): container finished" podID="8f76499e-97ad-4c03-8a72-a62f65619596" containerID="2b7e0cbea7cc9adfbc4ddedcd2bb9ab689149d004cf05ae4c38d9a2c2ec563f5" exitCode=0 Oct 30 18:34:16.756756 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.756729 6960 generic.go:356] "Generic (PLEG): container finished" podID="8f76499e-97ad-4c03-8a72-a62f65619596" containerID="a7ec88732cafa80ab354e8864f75c1833177871423f5ca944fbf24dc7bb44a2b" exitCode=0 Oct 30 18:34:16.756756 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.756718 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8f76499e-97ad-4c03-8a72-a62f65619596","Type":"ContainerDied","Data":"563244759dbdbc2c58d0b4ba9e94b736d2bb8e69feb6e845291e2e2d8bf3fcb2"} Oct 30 18:34:16.757344 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.756790 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:16.757344 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.756743 6960 generic.go:356] "Generic (PLEG): container finished" podID="8f76499e-97ad-4c03-8a72-a62f65619596" containerID="b049f7e3597139cac61bd3d2c74278bbfd9a2df99c6ffeaef14d0bda8e236eef" exitCode=0 Oct 30 18:34:16.757344 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.756810 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8f76499e-97ad-4c03-8a72-a62f65619596","Type":"ContainerDied","Data":"be397bdaf885f8c055c80395dea268c2f30e89f2cb1526720d2440af75217759"} Oct 30 18:34:16.757344 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.756824 6960 generic.go:356] "Generic (PLEG): container finished" podID="8f76499e-97ad-4c03-8a72-a62f65619596" containerID="c5aea0259e406d5b190a03fce661f9a4052cc4d10160e32c100ffdf12f7433d5" exitCode=0 Oct 30 18:34:16.757344 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.756839 6960 scope.go:122] "RemoveContainer" containerID="563244759dbdbc2c58d0b4ba9e94b736d2bb8e69feb6e845291e2e2d8bf3fcb2" Oct 30 18:34:16.757344 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.756860 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8f76499e-97ad-4c03-8a72-a62f65619596","Type":"ContainerDied","Data":"2b7e0cbea7cc9adfbc4ddedcd2bb9ab689149d004cf05ae4c38d9a2c2ec563f5"} Oct 30 18:34:16.757344 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.756902 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8f76499e-97ad-4c03-8a72-a62f65619596","Type":"ContainerDied","Data":"a7ec88732cafa80ab354e8864f75c1833177871423f5ca944fbf24dc7bb44a2b"} Oct 30 18:34:16.757344 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.756942 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8f76499e-97ad-4c03-8a72-a62f65619596","Type":"ContainerDied","Data":"b049f7e3597139cac61bd3d2c74278bbfd9a2df99c6ffeaef14d0bda8e236eef"} Oct 30 18:34:16.757344 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.756979 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8f76499e-97ad-4c03-8a72-a62f65619596","Type":"ContainerDied","Data":"c5aea0259e406d5b190a03fce661f9a4052cc4d10160e32c100ffdf12f7433d5"} Oct 30 18:34:16.757344 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.757004 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8f76499e-97ad-4c03-8a72-a62f65619596","Type":"ContainerDied","Data":"32522431fb329153a6eeebccc7680bd24dcdea651248be4cc3d5f50b8bb5a1e3"} Oct 30 18:34:16.758652 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.758623 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-85d6484797-5gnbj" event={"ID":"48000d54-fb98-4094-8160-b44be17f190c","Type":"ContainerStarted","Data":"95114c2305cb0ffaf88ad65468d21508270e13542c621bae7449885ee3ace247"} Oct 30 18:34:16.766061 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.766041 6960 scope.go:122] "RemoveContainer" containerID="be397bdaf885f8c055c80395dea268c2f30e89f2cb1526720d2440af75217759" Oct 30 18:34:16.773741 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.773719 6960 scope.go:122] "RemoveContainer" containerID="2b7e0cbea7cc9adfbc4ddedcd2bb9ab689149d004cf05ae4c38d9a2c2ec563f5" Oct 30 18:34:16.779269 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.779224 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 18:34:16.782424 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.782406 6960 scope.go:122] "RemoveContainer" containerID="a7ec88732cafa80ab354e8864f75c1833177871423f5ca944fbf24dc7bb44a2b" Oct 30 18:34:16.784134 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.784113 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 18:34:16.791712 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.791695 6960 scope.go:122] "RemoveContainer" containerID="b049f7e3597139cac61bd3d2c74278bbfd9a2df99c6ffeaef14d0bda8e236eef" Oct 30 18:34:16.800087 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.800065 6960 scope.go:122] "RemoveContainer" containerID="c5aea0259e406d5b190a03fce661f9a4052cc4d10160e32c100ffdf12f7433d5" Oct 30 18:34:16.808686 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.808668 6960 scope.go:122] "RemoveContainer" containerID="696e8840b46d4d1acf76b61bafc7bc0d953821760feec22065f848f3daa8eaf7" Oct 30 18:34:16.816407 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.816386 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 18:34:16.816700 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.816679 6960 scope.go:122] "RemoveContainer" containerID="563244759dbdbc2c58d0b4ba9e94b736d2bb8e69feb6e845291e2e2d8bf3fcb2" Oct 30 18:34:16.816890 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.816869 6960 cpu_manager.go:414] "containerMap: removing container" podUID="8f76499e-97ad-4c03-8a72-a62f65619596" containerName="init-config-reloader" Oct 30 18:34:16.816952 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.816893 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f76499e-97ad-4c03-8a72-a62f65619596" containerName="init-config-reloader" Oct 30 18:34:16.816952 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.816922 6960 cpu_manager.go:414] "containerMap: removing container" podUID="8f76499e-97ad-4c03-8a72-a62f65619596" containerName="kube-rbac-proxy-web" Oct 30 18:34:16.816952 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.816931 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f76499e-97ad-4c03-8a72-a62f65619596" containerName="kube-rbac-proxy-web" Oct 30 18:34:16.816952 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.816943 6960 cpu_manager.go:414] "containerMap: removing container" podUID="8f76499e-97ad-4c03-8a72-a62f65619596" containerName="config-reloader" Oct 30 18:34:16.816952 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.816950 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f76499e-97ad-4c03-8a72-a62f65619596" containerName="config-reloader" Oct 30 18:34:16.817122 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.816971 6960 cpu_manager.go:414] "containerMap: removing container" podUID="8f76499e-97ad-4c03-8a72-a62f65619596" containerName="kube-rbac-proxy-thanos" Oct 30 18:34:16.817122 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.816980 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f76499e-97ad-4c03-8a72-a62f65619596" containerName="kube-rbac-proxy-thanos" Oct 30 18:34:16.817122 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.816991 6960 cpu_manager.go:414] "containerMap: removing container" podUID="8f76499e-97ad-4c03-8a72-a62f65619596" containerName="kube-rbac-proxy" Oct 30 18:34:16.817122 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.816998 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f76499e-97ad-4c03-8a72-a62f65619596" containerName="kube-rbac-proxy" Oct 30 18:34:16.817122 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.817020 6960 cpu_manager.go:414] "containerMap: removing container" podUID="8f76499e-97ad-4c03-8a72-a62f65619596" containerName="thanos-sidecar" Oct 30 18:34:16.817122 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.817028 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f76499e-97ad-4c03-8a72-a62f65619596" containerName="thanos-sidecar" Oct 30 18:34:16.817122 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.817039 6960 cpu_manager.go:414] "containerMap: removing container" podUID="8f76499e-97ad-4c03-8a72-a62f65619596" containerName="prometheus" Oct 30 18:34:16.817122 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.817046 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f76499e-97ad-4c03-8a72-a62f65619596" containerName="prometheus" Oct 30 18:34:16.817122 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.817117 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f76499e-97ad-4c03-8a72-a62f65619596" containerName="init-config-reloader" Oct 30 18:34:16.817671 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.817130 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f76499e-97ad-4c03-8a72-a62f65619596" containerName="kube-rbac-proxy-web" Oct 30 18:34:16.817671 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.817156 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f76499e-97ad-4c03-8a72-a62f65619596" containerName="config-reloader" Oct 30 18:34:16.817671 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.817166 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f76499e-97ad-4c03-8a72-a62f65619596" containerName="kube-rbac-proxy-thanos" Oct 30 18:34:16.817671 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.817176 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f76499e-97ad-4c03-8a72-a62f65619596" containerName="kube-rbac-proxy" Oct 30 18:34:16.817671 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.817187 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f76499e-97ad-4c03-8a72-a62f65619596" containerName="thanos-sidecar" Oct 30 18:34:16.817671 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.817196 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f76499e-97ad-4c03-8a72-a62f65619596" containerName="prometheus" Oct 30 18:34:16.817671 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:34:16.817593 6960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"563244759dbdbc2c58d0b4ba9e94b736d2bb8e69feb6e845291e2e2d8bf3fcb2\": container with ID starting with 563244759dbdbc2c58d0b4ba9e94b736d2bb8e69feb6e845291e2e2d8bf3fcb2 not found: ID does not exist" containerID="563244759dbdbc2c58d0b4ba9e94b736d2bb8e69feb6e845291e2e2d8bf3fcb2" Oct 30 18:34:16.817671 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.817628 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"563244759dbdbc2c58d0b4ba9e94b736d2bb8e69feb6e845291e2e2d8bf3fcb2"} err="failed to get container status \"563244759dbdbc2c58d0b4ba9e94b736d2bb8e69feb6e845291e2e2d8bf3fcb2\": rpc error: code = NotFound desc = could not find container \"563244759dbdbc2c58d0b4ba9e94b736d2bb8e69feb6e845291e2e2d8bf3fcb2\": container with ID starting with 563244759dbdbc2c58d0b4ba9e94b736d2bb8e69feb6e845291e2e2d8bf3fcb2 not found: ID does not exist" Oct 30 18:34:16.817671 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.817662 6960 scope.go:122] "RemoveContainer" containerID="be397bdaf885f8c055c80395dea268c2f30e89f2cb1526720d2440af75217759" Oct 30 18:34:16.818025 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:34:16.817923 6960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be397bdaf885f8c055c80395dea268c2f30e89f2cb1526720d2440af75217759\": container with ID starting with be397bdaf885f8c055c80395dea268c2f30e89f2cb1526720d2440af75217759 not found: ID does not exist" containerID="be397bdaf885f8c055c80395dea268c2f30e89f2cb1526720d2440af75217759" Oct 30 18:34:16.818025 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.817957 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be397bdaf885f8c055c80395dea268c2f30e89f2cb1526720d2440af75217759"} err="failed to get container status \"be397bdaf885f8c055c80395dea268c2f30e89f2cb1526720d2440af75217759\": rpc error: code = NotFound desc = could not find container \"be397bdaf885f8c055c80395dea268c2f30e89f2cb1526720d2440af75217759\": container with ID starting with be397bdaf885f8c055c80395dea268c2f30e89f2cb1526720d2440af75217759 not found: ID does not exist" Oct 30 18:34:16.818025 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.817981 6960 scope.go:122] "RemoveContainer" containerID="2b7e0cbea7cc9adfbc4ddedcd2bb9ab689149d004cf05ae4c38d9a2c2ec563f5" Oct 30 18:34:16.818347 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:34:16.818234 6960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b7e0cbea7cc9adfbc4ddedcd2bb9ab689149d004cf05ae4c38d9a2c2ec563f5\": container with ID starting with 2b7e0cbea7cc9adfbc4ddedcd2bb9ab689149d004cf05ae4c38d9a2c2ec563f5 not found: ID does not exist" containerID="2b7e0cbea7cc9adfbc4ddedcd2bb9ab689149d004cf05ae4c38d9a2c2ec563f5" Oct 30 18:34:16.818347 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.818265 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b7e0cbea7cc9adfbc4ddedcd2bb9ab689149d004cf05ae4c38d9a2c2ec563f5"} err="failed to get container status \"2b7e0cbea7cc9adfbc4ddedcd2bb9ab689149d004cf05ae4c38d9a2c2ec563f5\": rpc error: code = NotFound desc = could not find container \"2b7e0cbea7cc9adfbc4ddedcd2bb9ab689149d004cf05ae4c38d9a2c2ec563f5\": container with ID starting with 2b7e0cbea7cc9adfbc4ddedcd2bb9ab689149d004cf05ae4c38d9a2c2ec563f5 not found: ID does not exist" Oct 30 18:34:16.818347 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.818282 6960 scope.go:122] "RemoveContainer" containerID="a7ec88732cafa80ab354e8864f75c1833177871423f5ca944fbf24dc7bb44a2b" Oct 30 18:34:16.818575 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:34:16.818552 6960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7ec88732cafa80ab354e8864f75c1833177871423f5ca944fbf24dc7bb44a2b\": container with ID starting with a7ec88732cafa80ab354e8864f75c1833177871423f5ca944fbf24dc7bb44a2b not found: ID does not exist" containerID="a7ec88732cafa80ab354e8864f75c1833177871423f5ca944fbf24dc7bb44a2b" Oct 30 18:34:16.818624 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.818582 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7ec88732cafa80ab354e8864f75c1833177871423f5ca944fbf24dc7bb44a2b"} err="failed to get container status \"a7ec88732cafa80ab354e8864f75c1833177871423f5ca944fbf24dc7bb44a2b\": rpc error: code = NotFound desc = could not find container \"a7ec88732cafa80ab354e8864f75c1833177871423f5ca944fbf24dc7bb44a2b\": container with ID starting with a7ec88732cafa80ab354e8864f75c1833177871423f5ca944fbf24dc7bb44a2b not found: ID does not exist" Oct 30 18:34:16.818624 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.818601 6960 scope.go:122] "RemoveContainer" containerID="b049f7e3597139cac61bd3d2c74278bbfd9a2df99c6ffeaef14d0bda8e236eef" Oct 30 18:34:16.818930 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:34:16.818910 6960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b049f7e3597139cac61bd3d2c74278bbfd9a2df99c6ffeaef14d0bda8e236eef\": container with ID starting with b049f7e3597139cac61bd3d2c74278bbfd9a2df99c6ffeaef14d0bda8e236eef not found: ID does not exist" containerID="b049f7e3597139cac61bd3d2c74278bbfd9a2df99c6ffeaef14d0bda8e236eef" Oct 30 18:34:16.819021 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.818932 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b049f7e3597139cac61bd3d2c74278bbfd9a2df99c6ffeaef14d0bda8e236eef"} err="failed to get container status \"b049f7e3597139cac61bd3d2c74278bbfd9a2df99c6ffeaef14d0bda8e236eef\": rpc error: code = NotFound desc = could not find container \"b049f7e3597139cac61bd3d2c74278bbfd9a2df99c6ffeaef14d0bda8e236eef\": container with ID starting with b049f7e3597139cac61bd3d2c74278bbfd9a2df99c6ffeaef14d0bda8e236eef not found: ID does not exist" Oct 30 18:34:16.819021 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.818945 6960 scope.go:122] "RemoveContainer" containerID="c5aea0259e406d5b190a03fce661f9a4052cc4d10160e32c100ffdf12f7433d5" Oct 30 18:34:16.819313 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:34:16.819292 6960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5aea0259e406d5b190a03fce661f9a4052cc4d10160e32c100ffdf12f7433d5\": container with ID starting with c5aea0259e406d5b190a03fce661f9a4052cc4d10160e32c100ffdf12f7433d5 not found: ID does not exist" containerID="c5aea0259e406d5b190a03fce661f9a4052cc4d10160e32c100ffdf12f7433d5" Oct 30 18:34:16.819383 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.819321 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5aea0259e406d5b190a03fce661f9a4052cc4d10160e32c100ffdf12f7433d5"} err="failed to get container status \"c5aea0259e406d5b190a03fce661f9a4052cc4d10160e32c100ffdf12f7433d5\": rpc error: code = NotFound desc = could not find container \"c5aea0259e406d5b190a03fce661f9a4052cc4d10160e32c100ffdf12f7433d5\": container with ID starting with c5aea0259e406d5b190a03fce661f9a4052cc4d10160e32c100ffdf12f7433d5 not found: ID does not exist" Oct 30 18:34:16.819383 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.819338 6960 scope.go:122] "RemoveContainer" containerID="696e8840b46d4d1acf76b61bafc7bc0d953821760feec22065f848f3daa8eaf7" Oct 30 18:34:16.819618 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:34:16.819600 6960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"696e8840b46d4d1acf76b61bafc7bc0d953821760feec22065f848f3daa8eaf7\": container with ID starting with 696e8840b46d4d1acf76b61bafc7bc0d953821760feec22065f848f3daa8eaf7 not found: ID does not exist" containerID="696e8840b46d4d1acf76b61bafc7bc0d953821760feec22065f848f3daa8eaf7" Oct 30 18:34:16.819692 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.819625 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"696e8840b46d4d1acf76b61bafc7bc0d953821760feec22065f848f3daa8eaf7"} err="failed to get container status \"696e8840b46d4d1acf76b61bafc7bc0d953821760feec22065f848f3daa8eaf7\": rpc error: code = NotFound desc = could not find container \"696e8840b46d4d1acf76b61bafc7bc0d953821760feec22065f848f3daa8eaf7\": container with ID starting with 696e8840b46d4d1acf76b61bafc7bc0d953821760feec22065f848f3daa8eaf7 not found: ID does not exist" Oct 30 18:34:16.819692 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.819644 6960 scope.go:122] "RemoveContainer" containerID="563244759dbdbc2c58d0b4ba9e94b736d2bb8e69feb6e845291e2e2d8bf3fcb2" Oct 30 18:34:16.819947 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.819927 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"563244759dbdbc2c58d0b4ba9e94b736d2bb8e69feb6e845291e2e2d8bf3fcb2"} err="failed to get container status \"563244759dbdbc2c58d0b4ba9e94b736d2bb8e69feb6e845291e2e2d8bf3fcb2\": rpc error: code = NotFound desc = could not find container \"563244759dbdbc2c58d0b4ba9e94b736d2bb8e69feb6e845291e2e2d8bf3fcb2\": container with ID starting with 563244759dbdbc2c58d0b4ba9e94b736d2bb8e69feb6e845291e2e2d8bf3fcb2 not found: ID does not exist" Oct 30 18:34:16.820021 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.819947 6960 scope.go:122] "RemoveContainer" containerID="be397bdaf885f8c055c80395dea268c2f30e89f2cb1526720d2440af75217759" Oct 30 18:34:16.820249 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.820224 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be397bdaf885f8c055c80395dea268c2f30e89f2cb1526720d2440af75217759"} err="failed to get container status \"be397bdaf885f8c055c80395dea268c2f30e89f2cb1526720d2440af75217759\": rpc error: code = NotFound desc = could not find container \"be397bdaf885f8c055c80395dea268c2f30e89f2cb1526720d2440af75217759\": container with ID starting with be397bdaf885f8c055c80395dea268c2f30e89f2cb1526720d2440af75217759 not found: ID does not exist" Oct 30 18:34:16.820338 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.820250 6960 scope.go:122] "RemoveContainer" containerID="2b7e0cbea7cc9adfbc4ddedcd2bb9ab689149d004cf05ae4c38d9a2c2ec563f5" Oct 30 18:34:16.820555 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.820529 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b7e0cbea7cc9adfbc4ddedcd2bb9ab689149d004cf05ae4c38d9a2c2ec563f5"} err="failed to get container status \"2b7e0cbea7cc9adfbc4ddedcd2bb9ab689149d004cf05ae4c38d9a2c2ec563f5\": rpc error: code = NotFound desc = could not find container \"2b7e0cbea7cc9adfbc4ddedcd2bb9ab689149d004cf05ae4c38d9a2c2ec563f5\": container with ID starting with 2b7e0cbea7cc9adfbc4ddedcd2bb9ab689149d004cf05ae4c38d9a2c2ec563f5 not found: ID does not exist" Oct 30 18:34:16.820607 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.820557 6960 scope.go:122] "RemoveContainer" containerID="a7ec88732cafa80ab354e8864f75c1833177871423f5ca944fbf24dc7bb44a2b" Oct 30 18:34:16.820870 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.820847 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7ec88732cafa80ab354e8864f75c1833177871423f5ca944fbf24dc7bb44a2b"} err="failed to get container status \"a7ec88732cafa80ab354e8864f75c1833177871423f5ca944fbf24dc7bb44a2b\": rpc error: code = NotFound desc = could not find container \"a7ec88732cafa80ab354e8864f75c1833177871423f5ca944fbf24dc7bb44a2b\": container with ID starting with a7ec88732cafa80ab354e8864f75c1833177871423f5ca944fbf24dc7bb44a2b not found: ID does not exist" Oct 30 18:34:16.820931 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.820870 6960 scope.go:122] "RemoveContainer" containerID="b049f7e3597139cac61bd3d2c74278bbfd9a2df99c6ffeaef14d0bda8e236eef" Oct 30 18:34:16.821131 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.821109 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b049f7e3597139cac61bd3d2c74278bbfd9a2df99c6ffeaef14d0bda8e236eef"} err="failed to get container status \"b049f7e3597139cac61bd3d2c74278bbfd9a2df99c6ffeaef14d0bda8e236eef\": rpc error: code = NotFound desc = could not find container \"b049f7e3597139cac61bd3d2c74278bbfd9a2df99c6ffeaef14d0bda8e236eef\": container with ID starting with b049f7e3597139cac61bd3d2c74278bbfd9a2df99c6ffeaef14d0bda8e236eef not found: ID does not exist" Oct 30 18:34:16.821281 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.821134 6960 scope.go:122] "RemoveContainer" containerID="c5aea0259e406d5b190a03fce661f9a4052cc4d10160e32c100ffdf12f7433d5" Oct 30 18:34:16.821449 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.821426 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5aea0259e406d5b190a03fce661f9a4052cc4d10160e32c100ffdf12f7433d5"} err="failed to get container status \"c5aea0259e406d5b190a03fce661f9a4052cc4d10160e32c100ffdf12f7433d5\": rpc error: code = NotFound desc = could not find container \"c5aea0259e406d5b190a03fce661f9a4052cc4d10160e32c100ffdf12f7433d5\": container with ID starting with c5aea0259e406d5b190a03fce661f9a4052cc4d10160e32c100ffdf12f7433d5 not found: ID does not exist" Oct 30 18:34:16.821500 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.821451 6960 scope.go:122] "RemoveContainer" containerID="696e8840b46d4d1acf76b61bafc7bc0d953821760feec22065f848f3daa8eaf7" Oct 30 18:34:16.821729 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.821705 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"696e8840b46d4d1acf76b61bafc7bc0d953821760feec22065f848f3daa8eaf7"} err="failed to get container status \"696e8840b46d4d1acf76b61bafc7bc0d953821760feec22065f848f3daa8eaf7\": rpc error: code = NotFound desc = could not find container \"696e8840b46d4d1acf76b61bafc7bc0d953821760feec22065f848f3daa8eaf7\": container with ID starting with 696e8840b46d4d1acf76b61bafc7bc0d953821760feec22065f848f3daa8eaf7 not found: ID does not exist" Oct 30 18:34:16.821729 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.821729 6960 scope.go:122] "RemoveContainer" containerID="563244759dbdbc2c58d0b4ba9e94b736d2bb8e69feb6e845291e2e2d8bf3fcb2" Oct 30 18:34:16.821988 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.821967 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"563244759dbdbc2c58d0b4ba9e94b736d2bb8e69feb6e845291e2e2d8bf3fcb2"} err="failed to get container status \"563244759dbdbc2c58d0b4ba9e94b736d2bb8e69feb6e845291e2e2d8bf3fcb2\": rpc error: code = NotFound desc = could not find container \"563244759dbdbc2c58d0b4ba9e94b736d2bb8e69feb6e845291e2e2d8bf3fcb2\": container with ID starting with 563244759dbdbc2c58d0b4ba9e94b736d2bb8e69feb6e845291e2e2d8bf3fcb2 not found: ID does not exist" Oct 30 18:34:16.822068 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.821990 6960 scope.go:122] "RemoveContainer" containerID="be397bdaf885f8c055c80395dea268c2f30e89f2cb1526720d2440af75217759" Oct 30 18:34:16.822304 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.822285 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be397bdaf885f8c055c80395dea268c2f30e89f2cb1526720d2440af75217759"} err="failed to get container status \"be397bdaf885f8c055c80395dea268c2f30e89f2cb1526720d2440af75217759\": rpc error: code = NotFound desc = could not find container \"be397bdaf885f8c055c80395dea268c2f30e89f2cb1526720d2440af75217759\": container with ID starting with be397bdaf885f8c055c80395dea268c2f30e89f2cb1526720d2440af75217759 not found: ID does not exist" Oct 30 18:34:16.822304 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.822303 6960 scope.go:122] "RemoveContainer" containerID="2b7e0cbea7cc9adfbc4ddedcd2bb9ab689149d004cf05ae4c38d9a2c2ec563f5" Oct 30 18:34:16.822551 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.822527 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b7e0cbea7cc9adfbc4ddedcd2bb9ab689149d004cf05ae4c38d9a2c2ec563f5"} err="failed to get container status \"2b7e0cbea7cc9adfbc4ddedcd2bb9ab689149d004cf05ae4c38d9a2c2ec563f5\": rpc error: code = NotFound desc = could not find container \"2b7e0cbea7cc9adfbc4ddedcd2bb9ab689149d004cf05ae4c38d9a2c2ec563f5\": container with ID starting with 2b7e0cbea7cc9adfbc4ddedcd2bb9ab689149d004cf05ae4c38d9a2c2ec563f5 not found: ID does not exist" Oct 30 18:34:16.822625 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.822552 6960 scope.go:122] "RemoveContainer" containerID="a7ec88732cafa80ab354e8864f75c1833177871423f5ca944fbf24dc7bb44a2b" Oct 30 18:34:16.822625 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.822564 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:16.822879 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.822857 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7ec88732cafa80ab354e8864f75c1833177871423f5ca944fbf24dc7bb44a2b"} err="failed to get container status \"a7ec88732cafa80ab354e8864f75c1833177871423f5ca944fbf24dc7bb44a2b\": rpc error: code = NotFound desc = could not find container \"a7ec88732cafa80ab354e8864f75c1833177871423f5ca944fbf24dc7bb44a2b\": container with ID starting with a7ec88732cafa80ab354e8864f75c1833177871423f5ca944fbf24dc7bb44a2b not found: ID does not exist" Oct 30 18:34:16.822955 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.822880 6960 scope.go:122] "RemoveContainer" containerID="b049f7e3597139cac61bd3d2c74278bbfd9a2df99c6ffeaef14d0bda8e236eef" Oct 30 18:34:16.823264 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.823237 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b049f7e3597139cac61bd3d2c74278bbfd9a2df99c6ffeaef14d0bda8e236eef"} err="failed to get container status \"b049f7e3597139cac61bd3d2c74278bbfd9a2df99c6ffeaef14d0bda8e236eef\": rpc error: code = NotFound desc = could not find container \"b049f7e3597139cac61bd3d2c74278bbfd9a2df99c6ffeaef14d0bda8e236eef\": container with ID starting with b049f7e3597139cac61bd3d2c74278bbfd9a2df99c6ffeaef14d0bda8e236eef not found: ID does not exist" Oct 30 18:34:16.823361 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.823264 6960 scope.go:122] "RemoveContainer" containerID="c5aea0259e406d5b190a03fce661f9a4052cc4d10160e32c100ffdf12f7433d5" Oct 30 18:34:16.823540 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.823517 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5aea0259e406d5b190a03fce661f9a4052cc4d10160e32c100ffdf12f7433d5"} err="failed to get container status \"c5aea0259e406d5b190a03fce661f9a4052cc4d10160e32c100ffdf12f7433d5\": rpc error: code = NotFound desc = could not find container \"c5aea0259e406d5b190a03fce661f9a4052cc4d10160e32c100ffdf12f7433d5\": container with ID starting with c5aea0259e406d5b190a03fce661f9a4052cc4d10160e32c100ffdf12f7433d5 not found: ID does not exist" Oct 30 18:34:16.823609 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.823540 6960 scope.go:122] "RemoveContainer" containerID="696e8840b46d4d1acf76b61bafc7bc0d953821760feec22065f848f3daa8eaf7" Oct 30 18:34:16.823832 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.823810 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"696e8840b46d4d1acf76b61bafc7bc0d953821760feec22065f848f3daa8eaf7"} err="failed to get container status \"696e8840b46d4d1acf76b61bafc7bc0d953821760feec22065f848f3daa8eaf7\": rpc error: code = NotFound desc = could not find container \"696e8840b46d4d1acf76b61bafc7bc0d953821760feec22065f848f3daa8eaf7\": container with ID starting with 696e8840b46d4d1acf76b61bafc7bc0d953821760feec22065f848f3daa8eaf7 not found: ID does not exist" Oct 30 18:34:16.823875 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.823835 6960 scope.go:122] "RemoveContainer" containerID="563244759dbdbc2c58d0b4ba9e94b736d2bb8e69feb6e845291e2e2d8bf3fcb2" Oct 30 18:34:16.824375 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.824346 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"563244759dbdbc2c58d0b4ba9e94b736d2bb8e69feb6e845291e2e2d8bf3fcb2"} err="failed to get container status \"563244759dbdbc2c58d0b4ba9e94b736d2bb8e69feb6e845291e2e2d8bf3fcb2\": rpc error: code = NotFound desc = could not find container \"563244759dbdbc2c58d0b4ba9e94b736d2bb8e69feb6e845291e2e2d8bf3fcb2\": container with ID starting with 563244759dbdbc2c58d0b4ba9e94b736d2bb8e69feb6e845291e2e2d8bf3fcb2 not found: ID does not exist" Oct 30 18:34:16.824375 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.824374 6960 scope.go:122] "RemoveContainer" containerID="be397bdaf885f8c055c80395dea268c2f30e89f2cb1526720d2440af75217759" Oct 30 18:34:16.824698 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.824675 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be397bdaf885f8c055c80395dea268c2f30e89f2cb1526720d2440af75217759"} err="failed to get container status \"be397bdaf885f8c055c80395dea268c2f30e89f2cb1526720d2440af75217759\": rpc error: code = NotFound desc = could not find container \"be397bdaf885f8c055c80395dea268c2f30e89f2cb1526720d2440af75217759\": container with ID starting with be397bdaf885f8c055c80395dea268c2f30e89f2cb1526720d2440af75217759 not found: ID does not exist" Oct 30 18:34:16.824758 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.824700 6960 scope.go:122] "RemoveContainer" containerID="2b7e0cbea7cc9adfbc4ddedcd2bb9ab689149d004cf05ae4c38d9a2c2ec563f5" Oct 30 18:34:16.824972 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.824954 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b7e0cbea7cc9adfbc4ddedcd2bb9ab689149d004cf05ae4c38d9a2c2ec563f5"} err="failed to get container status \"2b7e0cbea7cc9adfbc4ddedcd2bb9ab689149d004cf05ae4c38d9a2c2ec563f5\": rpc error: code = NotFound desc = could not find container \"2b7e0cbea7cc9adfbc4ddedcd2bb9ab689149d004cf05ae4c38d9a2c2ec563f5\": container with ID starting with 2b7e0cbea7cc9adfbc4ddedcd2bb9ab689149d004cf05ae4c38d9a2c2ec563f5 not found: ID does not exist" Oct 30 18:34:16.824972 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.824971 6960 scope.go:122] "RemoveContainer" containerID="a7ec88732cafa80ab354e8864f75c1833177871423f5ca944fbf24dc7bb44a2b" Oct 30 18:34:16.825291 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.825270 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7ec88732cafa80ab354e8864f75c1833177871423f5ca944fbf24dc7bb44a2b"} err="failed to get container status \"a7ec88732cafa80ab354e8864f75c1833177871423f5ca944fbf24dc7bb44a2b\": rpc error: code = NotFound desc = could not find container \"a7ec88732cafa80ab354e8864f75c1833177871423f5ca944fbf24dc7bb44a2b\": container with ID starting with a7ec88732cafa80ab354e8864f75c1833177871423f5ca944fbf24dc7bb44a2b not found: ID does not exist" Oct 30 18:34:16.825371 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.825293 6960 scope.go:122] "RemoveContainer" containerID="b049f7e3597139cac61bd3d2c74278bbfd9a2df99c6ffeaef14d0bda8e236eef" Oct 30 18:34:16.825586 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.825565 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b049f7e3597139cac61bd3d2c74278bbfd9a2df99c6ffeaef14d0bda8e236eef"} err="failed to get container status \"b049f7e3597139cac61bd3d2c74278bbfd9a2df99c6ffeaef14d0bda8e236eef\": rpc error: code = NotFound desc = could not find container \"b049f7e3597139cac61bd3d2c74278bbfd9a2df99c6ffeaef14d0bda8e236eef\": container with ID starting with b049f7e3597139cac61bd3d2c74278bbfd9a2df99c6ffeaef14d0bda8e236eef not found: ID does not exist" Oct 30 18:34:16.825586 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.825584 6960 scope.go:122] "RemoveContainer" containerID="c5aea0259e406d5b190a03fce661f9a4052cc4d10160e32c100ffdf12f7433d5" Oct 30 18:34:16.825887 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.825868 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5aea0259e406d5b190a03fce661f9a4052cc4d10160e32c100ffdf12f7433d5"} err="failed to get container status \"c5aea0259e406d5b190a03fce661f9a4052cc4d10160e32c100ffdf12f7433d5\": rpc error: code = NotFound desc = could not find container \"c5aea0259e406d5b190a03fce661f9a4052cc4d10160e32c100ffdf12f7433d5\": container with ID starting with c5aea0259e406d5b190a03fce661f9a4052cc4d10160e32c100ffdf12f7433d5 not found: ID does not exist" Oct 30 18:34:16.825887 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.825887 6960 scope.go:122] "RemoveContainer" containerID="696e8840b46d4d1acf76b61bafc7bc0d953821760feec22065f848f3daa8eaf7" Oct 30 18:34:16.826184 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.826164 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"696e8840b46d4d1acf76b61bafc7bc0d953821760feec22065f848f3daa8eaf7"} err="failed to get container status \"696e8840b46d4d1acf76b61bafc7bc0d953821760feec22065f848f3daa8eaf7\": rpc error: code = NotFound desc = could not find container \"696e8840b46d4d1acf76b61bafc7bc0d953821760feec22065f848f3daa8eaf7\": container with ID starting with 696e8840b46d4d1acf76b61bafc7bc0d953821760feec22065f848f3daa8eaf7 not found: ID does not exist" Oct 30 18:34:16.826262 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.826184 6960 scope.go:122] "RemoveContainer" containerID="563244759dbdbc2c58d0b4ba9e94b736d2bb8e69feb6e845291e2e2d8bf3fcb2" Oct 30 18:34:16.826426 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.826400 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"563244759dbdbc2c58d0b4ba9e94b736d2bb8e69feb6e845291e2e2d8bf3fcb2"} err="failed to get container status \"563244759dbdbc2c58d0b4ba9e94b736d2bb8e69feb6e845291e2e2d8bf3fcb2\": rpc error: code = NotFound desc = could not find container \"563244759dbdbc2c58d0b4ba9e94b736d2bb8e69feb6e845291e2e2d8bf3fcb2\": container with ID starting with 563244759dbdbc2c58d0b4ba9e94b736d2bb8e69feb6e845291e2e2d8bf3fcb2 not found: ID does not exist" Oct 30 18:34:16.826525 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.826435 6960 scope.go:122] "RemoveContainer" containerID="be397bdaf885f8c055c80395dea268c2f30e89f2cb1526720d2440af75217759" Oct 30 18:34:16.827263 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.827235 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be397bdaf885f8c055c80395dea268c2f30e89f2cb1526720d2440af75217759"} err="failed to get container status \"be397bdaf885f8c055c80395dea268c2f30e89f2cb1526720d2440af75217759\": rpc error: code = NotFound desc = could not find container \"be397bdaf885f8c055c80395dea268c2f30e89f2cb1526720d2440af75217759\": container with ID starting with be397bdaf885f8c055c80395dea268c2f30e89f2cb1526720d2440af75217759 not found: ID does not exist" Oct 30 18:34:16.827263 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.827261 6960 scope.go:122] "RemoveContainer" containerID="2b7e0cbea7cc9adfbc4ddedcd2bb9ab689149d004cf05ae4c38d9a2c2ec563f5" Oct 30 18:34:16.827686 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.827657 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b7e0cbea7cc9adfbc4ddedcd2bb9ab689149d004cf05ae4c38d9a2c2ec563f5"} err="failed to get container status \"2b7e0cbea7cc9adfbc4ddedcd2bb9ab689149d004cf05ae4c38d9a2c2ec563f5\": rpc error: code = NotFound desc = could not find container \"2b7e0cbea7cc9adfbc4ddedcd2bb9ab689149d004cf05ae4c38d9a2c2ec563f5\": container with ID starting with 2b7e0cbea7cc9adfbc4ddedcd2bb9ab689149d004cf05ae4c38d9a2c2ec563f5 not found: ID does not exist" Oct 30 18:34:16.827764 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.827687 6960 scope.go:122] "RemoveContainer" containerID="a7ec88732cafa80ab354e8864f75c1833177871423f5ca944fbf24dc7bb44a2b" Oct 30 18:34:16.827957 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.827932 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7ec88732cafa80ab354e8864f75c1833177871423f5ca944fbf24dc7bb44a2b"} err="failed to get container status \"a7ec88732cafa80ab354e8864f75c1833177871423f5ca944fbf24dc7bb44a2b\": rpc error: code = NotFound desc = could not find container \"a7ec88732cafa80ab354e8864f75c1833177871423f5ca944fbf24dc7bb44a2b\": container with ID starting with a7ec88732cafa80ab354e8864f75c1833177871423f5ca944fbf24dc7bb44a2b not found: ID does not exist" Oct 30 18:34:16.828058 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.827958 6960 scope.go:122] "RemoveContainer" containerID="b049f7e3597139cac61bd3d2c74278bbfd9a2df99c6ffeaef14d0bda8e236eef" Oct 30 18:34:16.828264 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.828235 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b049f7e3597139cac61bd3d2c74278bbfd9a2df99c6ffeaef14d0bda8e236eef"} err="failed to get container status \"b049f7e3597139cac61bd3d2c74278bbfd9a2df99c6ffeaef14d0bda8e236eef\": rpc error: code = NotFound desc = could not find container \"b049f7e3597139cac61bd3d2c74278bbfd9a2df99c6ffeaef14d0bda8e236eef\": container with ID starting with b049f7e3597139cac61bd3d2c74278bbfd9a2df99c6ffeaef14d0bda8e236eef not found: ID does not exist" Oct 30 18:34:16.828352 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.828264 6960 scope.go:122] "RemoveContainer" containerID="c5aea0259e406d5b190a03fce661f9a4052cc4d10160e32c100ffdf12f7433d5" Oct 30 18:34:16.828587 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.828565 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5aea0259e406d5b190a03fce661f9a4052cc4d10160e32c100ffdf12f7433d5"} err="failed to get container status \"c5aea0259e406d5b190a03fce661f9a4052cc4d10160e32c100ffdf12f7433d5\": rpc error: code = NotFound desc = could not find container \"c5aea0259e406d5b190a03fce661f9a4052cc4d10160e32c100ffdf12f7433d5\": container with ID starting with c5aea0259e406d5b190a03fce661f9a4052cc4d10160e32c100ffdf12f7433d5 not found: ID does not exist" Oct 30 18:34:16.828587 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.828586 6960 scope.go:122] "RemoveContainer" containerID="696e8840b46d4d1acf76b61bafc7bc0d953821760feec22065f848f3daa8eaf7" Oct 30 18:34:16.828810 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.828788 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 18:34:16.828810 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.828807 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 18:34:16.828884 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.828877 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 18:34:16.828914 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.828888 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"696e8840b46d4d1acf76b61bafc7bc0d953821760feec22065f848f3daa8eaf7"} err="failed to get container status \"696e8840b46d4d1acf76b61bafc7bc0d953821760feec22065f848f3daa8eaf7\": rpc error: code = NotFound desc = could not find container \"696e8840b46d4d1acf76b61bafc7bc0d953821760feec22065f848f3daa8eaf7\": container with ID starting with 696e8840b46d4d1acf76b61bafc7bc0d953821760feec22065f848f3daa8eaf7 not found: ID does not exist" Oct 30 18:34:16.828949 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.828910 6960 scope.go:122] "RemoveContainer" containerID="563244759dbdbc2c58d0b4ba9e94b736d2bb8e69feb6e845291e2e2d8bf3fcb2" Oct 30 18:34:16.829007 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.828994 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 18:34:16.829471 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.829430 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 18:34:16.829471 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.829445 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 18:34:16.829609 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.829548 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"563244759dbdbc2c58d0b4ba9e94b736d2bb8e69feb6e845291e2e2d8bf3fcb2"} err="failed to get container status \"563244759dbdbc2c58d0b4ba9e94b736d2bb8e69feb6e845291e2e2d8bf3fcb2\": rpc error: code = NotFound desc = could not find container \"563244759dbdbc2c58d0b4ba9e94b736d2bb8e69feb6e845291e2e2d8bf3fcb2\": container with ID starting with 563244759dbdbc2c58d0b4ba9e94b736d2bb8e69feb6e845291e2e2d8bf3fcb2 not found: ID does not exist" Oct 30 18:34:16.829609 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.829568 6960 scope.go:122] "RemoveContainer" containerID="be397bdaf885f8c055c80395dea268c2f30e89f2cb1526720d2440af75217759" Oct 30 18:34:16.829790 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.829741 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 18:34:16.829790 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.829750 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-8b4gujb9r0sdv\"" Oct 30 18:34:16.829790 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.829762 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 18:34:16.829790 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.829743 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 18:34:16.830015 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.829868 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-5vglk\"" Oct 30 18:34:16.830015 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.829908 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 18:34:16.830015 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.829985 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 18:34:16.830015 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.830004 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be397bdaf885f8c055c80395dea268c2f30e89f2cb1526720d2440af75217759"} err="failed to get container status \"be397bdaf885f8c055c80395dea268c2f30e89f2cb1526720d2440af75217759\": rpc error: code = NotFound desc = could not find container \"be397bdaf885f8c055c80395dea268c2f30e89f2cb1526720d2440af75217759\": container with ID starting with be397bdaf885f8c055c80395dea268c2f30e89f2cb1526720d2440af75217759 not found: ID does not exist" Oct 30 18:34:16.830182 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.830026 6960 scope.go:122] "RemoveContainer" containerID="2b7e0cbea7cc9adfbc4ddedcd2bb9ab689149d004cf05ae4c38d9a2c2ec563f5" Oct 30 18:34:16.830182 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.830074 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 18:34:16.830367 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.830342 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b7e0cbea7cc9adfbc4ddedcd2bb9ab689149d004cf05ae4c38d9a2c2ec563f5"} err="failed to get container status \"2b7e0cbea7cc9adfbc4ddedcd2bb9ab689149d004cf05ae4c38d9a2c2ec563f5\": rpc error: code = NotFound desc = could not find container \"2b7e0cbea7cc9adfbc4ddedcd2bb9ab689149d004cf05ae4c38d9a2c2ec563f5\": container with ID starting with 2b7e0cbea7cc9adfbc4ddedcd2bb9ab689149d004cf05ae4c38d9a2c2ec563f5 not found: ID does not exist" Oct 30 18:34:16.830431 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.830371 6960 scope.go:122] "RemoveContainer" containerID="a7ec88732cafa80ab354e8864f75c1833177871423f5ca944fbf24dc7bb44a2b" Oct 30 18:34:16.835453 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.831418 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7ec88732cafa80ab354e8864f75c1833177871423f5ca944fbf24dc7bb44a2b"} err="failed to get container status \"a7ec88732cafa80ab354e8864f75c1833177871423f5ca944fbf24dc7bb44a2b\": rpc error: code = NotFound desc = could not find container \"a7ec88732cafa80ab354e8864f75c1833177871423f5ca944fbf24dc7bb44a2b\": container with ID starting with a7ec88732cafa80ab354e8864f75c1833177871423f5ca944fbf24dc7bb44a2b not found: ID does not exist" Oct 30 18:34:16.835453 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.831464 6960 scope.go:122] "RemoveContainer" containerID="b049f7e3597139cac61bd3d2c74278bbfd9a2df99c6ffeaef14d0bda8e236eef" Oct 30 18:34:16.835453 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.832278 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b049f7e3597139cac61bd3d2c74278bbfd9a2df99c6ffeaef14d0bda8e236eef"} err="failed to get container status \"b049f7e3597139cac61bd3d2c74278bbfd9a2df99c6ffeaef14d0bda8e236eef\": rpc error: code = NotFound desc = could not find container \"b049f7e3597139cac61bd3d2c74278bbfd9a2df99c6ffeaef14d0bda8e236eef\": container with ID starting with b049f7e3597139cac61bd3d2c74278bbfd9a2df99c6ffeaef14d0bda8e236eef not found: ID does not exist" Oct 30 18:34:16.835453 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.832304 6960 scope.go:122] "RemoveContainer" containerID="c5aea0259e406d5b190a03fce661f9a4052cc4d10160e32c100ffdf12f7433d5" Oct 30 18:34:16.835453 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.832644 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5aea0259e406d5b190a03fce661f9a4052cc4d10160e32c100ffdf12f7433d5"} err="failed to get container status \"c5aea0259e406d5b190a03fce661f9a4052cc4d10160e32c100ffdf12f7433d5\": rpc error: code = NotFound desc = could not find container \"c5aea0259e406d5b190a03fce661f9a4052cc4d10160e32c100ffdf12f7433d5\": container with ID starting with c5aea0259e406d5b190a03fce661f9a4052cc4d10160e32c100ffdf12f7433d5 not found: ID does not exist" Oct 30 18:34:16.835453 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.832666 6960 scope.go:122] "RemoveContainer" containerID="696e8840b46d4d1acf76b61bafc7bc0d953821760feec22065f848f3daa8eaf7" Oct 30 18:34:16.835453 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.832929 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"696e8840b46d4d1acf76b61bafc7bc0d953821760feec22065f848f3daa8eaf7"} err="failed to get container status \"696e8840b46d4d1acf76b61bafc7bc0d953821760feec22065f848f3daa8eaf7\": rpc error: code = NotFound desc = could not find container \"696e8840b46d4d1acf76b61bafc7bc0d953821760feec22065f848f3daa8eaf7\": container with ID starting with 696e8840b46d4d1acf76b61bafc7bc0d953821760feec22065f848f3daa8eaf7 not found: ID does not exist" Oct 30 18:34:16.835453 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.834494 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 18:34:16.835453 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.834674 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 18:34:16.841958 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.841936 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 18:34:16.965495 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.965456 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:16.965655 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.965508 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:16.965655 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.965626 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:16.965723 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.965655 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72-config-out\") pod \"prometheus-k8s-0\" (UID: \"fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:16.965723 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.965692 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:16.965723 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.965715 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:16.965814 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.965768 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:16.965814 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.965793 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:16.965875 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.965821 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:16.965875 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.965860 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:16.965933 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.965875 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:16.965933 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.965894 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72-config\") pod \"prometheus-k8s-0\" (UID: \"fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:16.965933 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.965916 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:16.966022 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.965952 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:16.966022 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.965979 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:16.966022 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.965994 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72-web-config\") pod \"prometheus-k8s-0\" (UID: \"fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:16.966108 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.966010 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w5pv\" (UniqueName: \"kubernetes.io/projected/fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72-kube-api-access-4w5pv\") pod \"prometheus-k8s-0\" (UID: \"fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:16.966108 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.966057 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:16.966108 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.966072 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:16.966108 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:16.966096 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:17.067089 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:17.067053 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:17.067261 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:17.067103 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:17.067261 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:17.067158 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:17.067261 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:17.067200 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:17.067261 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:17.067231 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:17.067447 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:17.067271 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:17.067447 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:17.067296 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72-config\") pod \"prometheus-k8s-0\" (UID: \"fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:17.067447 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:17.067318 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:17.067447 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:17.067333 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:17.067447 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:17.067359 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:17.067447 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:17.067372 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72-web-config\") pod \"prometheus-k8s-0\" (UID: \"fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:17.067447 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:17.067385 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w5pv\" (UniqueName: \"kubernetes.io/projected/fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72-kube-api-access-4w5pv\") pod \"prometheus-k8s-0\" (UID: \"fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:17.067447 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:17.067402 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:17.067447 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:17.067417 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:17.067447 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:17.067434 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:17.067917 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:17.067456 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:17.067917 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:17.067475 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:17.067917 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:17.067494 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:17.067917 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:17.067509 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72-config-out\") pod \"prometheus-k8s-0\" (UID: \"fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:17.067917 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:17.067523 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:17.067917 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:17.067876 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:17.068421 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:17.068266 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:17.068851 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:17.068826 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:17.068943 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:17.068913 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:17.069304 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:17.069273 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:17.070254 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:17.070227 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:17.070471 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:17.070444 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:17.071190 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:17.071107 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:17.071190 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:17.071447 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:17.071190 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:17.071629 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:17.071190 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:17.071666 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72-config\") pod \"prometheus-k8s-0\" (UID: \"fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:17.071190 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:17.071941 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:17.071190 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:17.072083 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72-config-out\") pod \"prometheus-k8s-0\" (UID: \"fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:17.072577 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:17.072431 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:17.073103 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:17.072892 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:17.074057 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:17.074039 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:17.074365 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:17.074347 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:17.074507 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:17.074489 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:17.074906 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:17.074890 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72-web-config\") pod \"prometheus-k8s-0\" (UID: \"fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:17.081437 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:17.081412 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w5pv\" (UniqueName: \"kubernetes.io/projected/fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72-kube-api-access-4w5pv\") pod \"prometheus-k8s-0\" (UID: \"fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:17.156280 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:17.156254 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:17.445230 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:17.445204 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 18:34:17.448004 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:34:17.447965 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc1c29c1_a5b4_4ea2_8a3f_e757894b2a72.slice/crio-d21a7dd8ea747de456de2719b8d94d7b533492338a87dd4bdfedc93e7182c078 WatchSource:0}: Error finding container d21a7dd8ea747de456de2719b8d94d7b533492338a87dd4bdfedc93e7182c078: Status 404 returned error can't find the container with id d21a7dd8ea747de456de2719b8d94d7b533492338a87dd4bdfedc93e7182c078 Oct 30 18:34:17.766168 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:17.766131 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-85d6484797-5gnbj" event={"ID":"48000d54-fb98-4094-8160-b44be17f190c","Type":"ContainerStarted","Data":"c7a6c89d4c5f772ee0463b70ddef9e19559136e0fac6ec146a840ba50304c906"} Oct 30 18:34:17.767657 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:17.767635 6960 generic.go:356] "Generic (PLEG): container finished" podID="fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72" containerID="30d8e505f4bc8813d09f90d5c9462faa0af9be948f83daf0d8a9683bf6ed9a60" exitCode=0 Oct 30 18:34:17.767742 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:17.767694 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72","Type":"ContainerDied","Data":"30d8e505f4bc8813d09f90d5c9462faa0af9be948f83daf0d8a9683bf6ed9a60"} Oct 30 18:34:17.767779 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:17.767768 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72","Type":"ContainerStarted","Data":"d21a7dd8ea747de456de2719b8d94d7b533492338a87dd4bdfedc93e7182c078"} Oct 30 18:34:18.590594 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:18.590562 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f76499e-97ad-4c03-8a72-a62f65619596" path="/var/lib/kubelet/pods/8f76499e-97ad-4c03-8a72-a62f65619596/volumes" Oct 30 18:34:18.772902 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:18.772869 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-85d6484797-5gnbj" event={"ID":"48000d54-fb98-4094-8160-b44be17f190c","Type":"ContainerStarted","Data":"6caf3a72ecf2473f154f6b3d1dfef90a906c9901d3334c36ed1698165f9f5832"} Oct 30 18:34:18.773305 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:18.772927 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-85d6484797-5gnbj" event={"ID":"48000d54-fb98-4094-8160-b44be17f190c","Type":"ContainerStarted","Data":"e85cfecc825c73211074264535b8af48ee8c76a2ac4dce8f4b2fb8a9d826f82c"} Oct 30 18:34:18.776340 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:18.776312 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72","Type":"ContainerStarted","Data":"c1d46dc001b301438c14f9ebc093dc53fa8a4f27a013d8cc231027b12dc3c66e"} Oct 30 18:34:18.776467 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:18.776382 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72","Type":"ContainerStarted","Data":"1101ba15b64997707e72833323a7bd741e6ad3842303602f6bb9619604bb4fd3"} Oct 30 18:34:18.776467 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:18.776426 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72","Type":"ContainerStarted","Data":"8b2b8c47b2d80bbb52d3fb12e3d95a20650526b09c199a29d2563fa634ba7457"} Oct 30 18:34:18.776530 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:18.776465 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72","Type":"ContainerStarted","Data":"8122f0dfca4e5a674466beb04c93e2a861cdf1efa11fdd9af4fb2e193db78d8a"} Oct 30 18:34:18.776530 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:18.776502 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72","Type":"ContainerStarted","Data":"9c61eee89507e753d2202e7fa61cacc6cc85d26815be0e3049eb9c31b971abf9"} Oct 30 18:34:18.776584 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:18.776541 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72","Type":"ContainerStarted","Data":"43e5211f1b2ecc2719da0df837d74faae8538ee224a4b55f73ba7ab291bf988b"} Oct 30 18:34:18.798713 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:18.798667 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-85d6484797-5gnbj" podStartSLOduration=2.275806599 podStartE2EDuration="3.798650526s" totalImagesPullingTime="1.522843927s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:34:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:34:18.797706947 +0000 UTC m=+26.326738085" watchObservedRunningTime="2026-10-30 18:34:18.798650526 +0000 UTC m=+26.327681665" Oct 30 18:34:18.831786 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:18.831751 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.812420757 podStartE2EDuration="2.831741321s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.320564ms" isStatefulPod=false podCreationTimestamp="2026-10-30 18:34:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:34:18.831515294 +0000 UTC m=+26.360546504" watchObservedRunningTime="2026-10-30 18:34:18.831741321 +0000 UTC m=+26.360772456" Oct 30 18:34:18.983611 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:18.983531 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:19.090599 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:19.090567 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:19.369932 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:19.369893 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:19.473313 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:19.473270 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:22.157197 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:22.157168 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:23.930065 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:34:23.930027 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 18:34:26.381257 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:26.381225 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-26f57" Oct 30 18:34:32.600807 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:32.600778 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:32.616673 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:32.616652 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:32.844845 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:32.844821 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 18:34:35.775691 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:35.775661 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:35.885353 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:35.885320 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:41.179777 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:41.179741 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:41.285331 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:41.285283 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:48.422958 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:48.422925 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:48.527024 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:48.526990 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:49.468743 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:49.468702 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:53.322518 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:53.322474 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:53.426453 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:53.426418 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:53.930967 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:34:53.930932 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 18:34:55.312041 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:55.311997 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:55.416752 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:55.416722 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:55.668530 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:55.668418 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:55.774499 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:34:55.774471 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:35:01.304549 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:35:01.304515 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:35:01.406648 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:35:01.406614 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:35:05.439906 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:35:05.439868 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:35:23.930699 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:35:23.930659 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 18:35:49.527257 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:35:49.527162 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:35:53.930414 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:35:53.930375 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 18:36:23.931097 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:36:23.931057 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 18:36:49.602616 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:36:49.602570 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:53.930317 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:36:53.930281 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 18:36:57.694339 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:36:57.694305 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:57.958939 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:36:57.958842 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:58.152686 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:36:58.152647 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:59.860519 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:36:59.860493 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:00.036510 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:00.036466 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:01.601554 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:01.601495 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:05.175331 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:05.175246 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:05.305778 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:05.305743 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:05.639137 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:05.639106 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:05.699853 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:05.699821 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-kdqdg"] Oct 30 18:37:05.703781 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:05.703755 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-kdqdg" Oct 30 18:37:05.715435 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:05.715412 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 18:37:05.725360 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:05.725340 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-xqpmm\"" Oct 30 18:37:05.753705 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:05.753680 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-kdqdg"] Oct 30 18:37:05.796135 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:05.796103 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:05.803162 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:05.803115 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6768ac2-1740-48aa-9305-0b3884c1ebf2-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-kdqdg\" (UID: \"f6768ac2-1740-48aa-9305-0b3884c1ebf2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-kdqdg" Oct 30 18:37:05.803268 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:05.803211 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6768ac2-1740-48aa-9305-0b3884c1ebf2-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-kdqdg\" (UID: \"f6768ac2-1740-48aa-9305-0b3884c1ebf2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-kdqdg" Oct 30 18:37:05.803268 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:05.803245 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6768ac2-1740-48aa-9305-0b3884c1ebf2-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-kdqdg\" (UID: \"f6768ac2-1740-48aa-9305-0b3884c1ebf2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-kdqdg" Oct 30 18:37:05.803364 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:05.803335 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6768ac2-1740-48aa-9305-0b3884c1ebf2-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-kdqdg\" (UID: \"f6768ac2-1740-48aa-9305-0b3884c1ebf2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-kdqdg" Oct 30 18:37:05.803410 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:05.803399 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6768ac2-1740-48aa-9305-0b3884c1ebf2-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-kdqdg\" (UID: \"f6768ac2-1740-48aa-9305-0b3884c1ebf2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-kdqdg" Oct 30 18:37:05.803450 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:05.803435 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ddnq\" (UniqueName: \"kubernetes.io/projected/f6768ac2-1740-48aa-9305-0b3884c1ebf2-kube-api-access-7ddnq\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-kdqdg\" (UID: \"f6768ac2-1740-48aa-9305-0b3884c1ebf2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-kdqdg" Oct 30 18:37:05.803606 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:05.803588 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6768ac2-1740-48aa-9305-0b3884c1ebf2-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-kdqdg\" (UID: \"f6768ac2-1740-48aa-9305-0b3884c1ebf2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-kdqdg" Oct 30 18:37:05.803644 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:05.803623 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6768ac2-1740-48aa-9305-0b3884c1ebf2-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-kdqdg\" (UID: \"f6768ac2-1740-48aa-9305-0b3884c1ebf2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-kdqdg" Oct 30 18:37:05.803698 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:05.803684 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6768ac2-1740-48aa-9305-0b3884c1ebf2-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-kdqdg\" (UID: \"f6768ac2-1740-48aa-9305-0b3884c1ebf2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-kdqdg" Oct 30 18:37:05.814316 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:05.814290 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 18:37:05.814316 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:05.814309 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 18:37:05.904357 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:05.904305 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6768ac2-1740-48aa-9305-0b3884c1ebf2-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-kdqdg\" (UID: \"f6768ac2-1740-48aa-9305-0b3884c1ebf2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-kdqdg" Oct 30 18:37:05.904357 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:05.904337 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ddnq\" (UniqueName: \"kubernetes.io/projected/f6768ac2-1740-48aa-9305-0b3884c1ebf2-kube-api-access-7ddnq\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-kdqdg\" (UID: \"f6768ac2-1740-48aa-9305-0b3884c1ebf2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-kdqdg" Oct 30 18:37:05.904521 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:05.904360 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6768ac2-1740-48aa-9305-0b3884c1ebf2-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-kdqdg\" (UID: \"f6768ac2-1740-48aa-9305-0b3884c1ebf2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-kdqdg" Oct 30 18:37:05.904521 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:05.904414 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6768ac2-1740-48aa-9305-0b3884c1ebf2-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-kdqdg\" (UID: \"f6768ac2-1740-48aa-9305-0b3884c1ebf2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-kdqdg" Oct 30 18:37:05.904521 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:05.904472 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6768ac2-1740-48aa-9305-0b3884c1ebf2-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-kdqdg\" (UID: \"f6768ac2-1740-48aa-9305-0b3884c1ebf2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-kdqdg" Oct 30 18:37:05.904521 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:05.904516 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6768ac2-1740-48aa-9305-0b3884c1ebf2-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-kdqdg\" (UID: \"f6768ac2-1740-48aa-9305-0b3884c1ebf2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-kdqdg" Oct 30 18:37:05.904748 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:05.904553 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6768ac2-1740-48aa-9305-0b3884c1ebf2-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-kdqdg\" (UID: \"f6768ac2-1740-48aa-9305-0b3884c1ebf2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-kdqdg" Oct 30 18:37:05.904748 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:05.904577 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6768ac2-1740-48aa-9305-0b3884c1ebf2-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-kdqdg\" (UID: \"f6768ac2-1740-48aa-9305-0b3884c1ebf2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-kdqdg" Oct 30 18:37:05.904748 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:05.904613 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6768ac2-1740-48aa-9305-0b3884c1ebf2-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-kdqdg\" (UID: \"f6768ac2-1740-48aa-9305-0b3884c1ebf2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-kdqdg" Oct 30 18:37:05.904748 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:05.904706 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6768ac2-1740-48aa-9305-0b3884c1ebf2-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-kdqdg\" (UID: \"f6768ac2-1740-48aa-9305-0b3884c1ebf2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-kdqdg" Oct 30 18:37:05.904970 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:05.904850 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6768ac2-1740-48aa-9305-0b3884c1ebf2-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-kdqdg\" (UID: \"f6768ac2-1740-48aa-9305-0b3884c1ebf2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-kdqdg" Oct 30 18:37:05.904970 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:05.904881 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6768ac2-1740-48aa-9305-0b3884c1ebf2-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-kdqdg\" (UID: \"f6768ac2-1740-48aa-9305-0b3884c1ebf2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-kdqdg" Oct 30 18:37:05.905062 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:05.904981 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6768ac2-1740-48aa-9305-0b3884c1ebf2-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-kdqdg\" (UID: \"f6768ac2-1740-48aa-9305-0b3884c1ebf2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-kdqdg" Oct 30 18:37:05.905294 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:05.905275 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6768ac2-1740-48aa-9305-0b3884c1ebf2-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-kdqdg\" (UID: \"f6768ac2-1740-48aa-9305-0b3884c1ebf2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-kdqdg" Oct 30 18:37:05.906871 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:05.906847 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6768ac2-1740-48aa-9305-0b3884c1ebf2-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-kdqdg\" (UID: \"f6768ac2-1740-48aa-9305-0b3884c1ebf2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-kdqdg" Oct 30 18:37:05.907016 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:05.907000 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6768ac2-1740-48aa-9305-0b3884c1ebf2-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-kdqdg\" (UID: \"f6768ac2-1740-48aa-9305-0b3884c1ebf2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-kdqdg" Oct 30 18:37:05.917001 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:05.916981 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446946k"] Oct 30 18:37:05.920653 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:05.920635 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446946k" Oct 30 18:37:05.924048 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:05.924025 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ddnq\" (UniqueName: \"kubernetes.io/projected/f6768ac2-1740-48aa-9305-0b3884c1ebf2-kube-api-access-7ddnq\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-kdqdg\" (UID: \"f6768ac2-1740-48aa-9305-0b3884c1ebf2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-kdqdg" Oct 30 18:37:05.924926 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:05.924904 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6768ac2-1740-48aa-9305-0b3884c1ebf2-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-9b4747bc-kdqdg\" (UID: \"f6768ac2-1740-48aa-9305-0b3884c1ebf2\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-kdqdg" Oct 30 18:37:05.927695 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:05.927675 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 18:37:05.927779 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:05.927769 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-vlnc7\"" Oct 30 18:37:05.928226 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:05.928208 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 18:37:05.947879 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:05.947857 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446946k"] Oct 30 18:37:05.951393 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:05.951349 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:06.005043 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:06.005022 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5d5b7017-2445-45a3-aa04-ba33d1bb268e-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446946k\" (UID: \"5d5b7017-2445-45a3-aa04-ba33d1bb268e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446946k" Oct 30 18:37:06.005202 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:06.005049 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5d5b7017-2445-45a3-aa04-ba33d1bb268e-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446946k\" (UID: \"5d5b7017-2445-45a3-aa04-ba33d1bb268e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446946k" Oct 30 18:37:06.005202 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:06.005189 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnkks\" (UniqueName: \"kubernetes.io/projected/5d5b7017-2445-45a3-aa04-ba33d1bb268e-kube-api-access-cnkks\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446946k\" (UID: \"5d5b7017-2445-45a3-aa04-ba33d1bb268e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446946k" Oct 30 18:37:06.018092 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:06.018071 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-kdqdg" Oct 30 18:37:06.106437 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:06.106412 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnkks\" (UniqueName: \"kubernetes.io/projected/5d5b7017-2445-45a3-aa04-ba33d1bb268e-kube-api-access-cnkks\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446946k\" (UID: \"5d5b7017-2445-45a3-aa04-ba33d1bb268e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446946k" Oct 30 18:37:06.106589 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:06.106464 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5d5b7017-2445-45a3-aa04-ba33d1bb268e-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446946k\" (UID: \"5d5b7017-2445-45a3-aa04-ba33d1bb268e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446946k" Oct 30 18:37:06.106589 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:06.106484 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5d5b7017-2445-45a3-aa04-ba33d1bb268e-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446946k\" (UID: \"5d5b7017-2445-45a3-aa04-ba33d1bb268e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446946k" Oct 30 18:37:06.106797 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:06.106780 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5d5b7017-2445-45a3-aa04-ba33d1bb268e-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446946k\" (UID: \"5d5b7017-2445-45a3-aa04-ba33d1bb268e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446946k" Oct 30 18:37:06.106834 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:06.106804 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5d5b7017-2445-45a3-aa04-ba33d1bb268e-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446946k\" (UID: \"5d5b7017-2445-45a3-aa04-ba33d1bb268e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446946k" Oct 30 18:37:06.118829 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:06.118811 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnkks\" (UniqueName: \"kubernetes.io/projected/5d5b7017-2445-45a3-aa04-ba33d1bb268e-kube-api-access-cnkks\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446946k\" (UID: \"5d5b7017-2445-45a3-aa04-ba33d1bb268e\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446946k" Oct 30 18:37:06.247583 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:06.247500 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446946k" Oct 30 18:37:06.293573 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:06.293469 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-kdqdg"] Oct 30 18:37:06.297197 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:37:06.297171 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6768ac2_1740_48aa_9305_0b3884c1ebf2.slice/crio-9d50d638d6f6cbf035ef3e117d95178f41d455c3a0bde32b9dd71d0f2dc462af WatchSource:0}: Error finding container 9d50d638d6f6cbf035ef3e117d95178f41d455c3a0bde32b9dd71d0f2dc462af: Status 404 returned error can't find the container with id 9d50d638d6f6cbf035ef3e117d95178f41d455c3a0bde32b9dd71d0f2dc462af Oct 30 18:37:06.385014 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:06.384981 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-kdqdg" event={"ID":"f6768ac2-1740-48aa-9305-0b3884c1ebf2","Type":"ContainerStarted","Data":"9d50d638d6f6cbf035ef3e117d95178f41d455c3a0bde32b9dd71d0f2dc462af"} Oct 30 18:37:06.527036 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:06.526957 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446946k"] Oct 30 18:37:06.530130 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:37:06.530107 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d5b7017_2445_45a3_aa04_ba33d1bb268e.slice/crio-021c35fc5be8ac59f6a960edbf4d6f2076f655c7248248785cb5bf3bb53fc20b WatchSource:0}: Error finding container 021c35fc5be8ac59f6a960edbf4d6f2076f655c7248248785cb5bf3bb53fc20b: Status 404 returned error can't find the container with id 021c35fc5be8ac59f6a960edbf4d6f2076f655c7248248785cb5bf3bb53fc20b Oct 30 18:37:07.392426 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:07.392379 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446946k" event={"ID":"5d5b7017-2445-45a3-aa04-ba33d1bb268e","Type":"ContainerStarted","Data":"021c35fc5be8ac59f6a960edbf4d6f2076f655c7248248785cb5bf3bb53fc20b"} Oct 30 18:37:08.884974 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:08.884938 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:09.644880 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:09.644843 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:37:09.644997 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:09.644927 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:37:09.644997 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:09.644973 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:37:10.409372 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:10.409332 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-kdqdg" event={"ID":"f6768ac2-1740-48aa-9305-0b3884c1ebf2","Type":"ContainerStarted","Data":"1b499a4ee7958ffe223a429821beefcf5c5eaf1d8606d590bc9ef035fcc489a0"} Oct 30 18:37:10.435011 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:10.434955 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-kdqdg" podStartSLOduration=2.091128337 podStartE2EDuration="5.434937535s" totalImagesPullingTime="3.343809198s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:37:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:37:10.432945712 +0000 UTC m=+197.961976862" watchObservedRunningTime="2026-10-30 18:37:10.434937535 +0000 UTC m=+197.963968667" Oct 30 18:37:11.018378 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:11.018337 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-kdqdg" Oct 30 18:37:11.023863 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:11.023838 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-kdqdg" Oct 30 18:37:11.414008 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:11.413970 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-kdqdg" Oct 30 18:37:11.415182 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:11.415138 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-9b4747bc-kdqdg" Oct 30 18:37:12.418680 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:12.418653 6960 generic.go:356] "Generic (PLEG): container finished" podID="5d5b7017-2445-45a3-aa04-ba33d1bb268e" containerID="d819a9384565235d078491e1efa69bbc55d99acb44bec4c9eea3fd975561c379" exitCode=0 Oct 30 18:37:12.419012 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:12.418735 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446946k" event={"ID":"5d5b7017-2445-45a3-aa04-ba33d1bb268e","Type":"ContainerDied","Data":"d819a9384565235d078491e1efa69bbc55d99acb44bec4c9eea3fd975561c379"} Oct 30 18:37:13.536598 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:13.536469 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:15.432626 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:15.432593 6960 generic.go:356] "Generic (PLEG): container finished" podID="5d5b7017-2445-45a3-aa04-ba33d1bb268e" containerID="fcfc67b9b8d3e1d85dbb13377f0975b9464c09b1d0298515c60eb9a2e25c1b0c" exitCode=0 Oct 30 18:37:15.432999 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:15.432677 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446946k" event={"ID":"5d5b7017-2445-45a3-aa04-ba33d1bb268e","Type":"ContainerDied","Data":"fcfc67b9b8d3e1d85dbb13377f0975b9464c09b1d0298515c60eb9a2e25c1b0c"} Oct 30 18:37:21.465306 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:21.465227 6960 generic.go:356] "Generic (PLEG): container finished" podID="5d5b7017-2445-45a3-aa04-ba33d1bb268e" containerID="21d6cf7b406a6ff9cedd8ae6f8cd1cc4ac70cf199200ab7265de25b17187aeb7" exitCode=0 Oct 30 18:37:21.465306 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:21.465269 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446946k" event={"ID":"5d5b7017-2445-45a3-aa04-ba33d1bb268e","Type":"ContainerDied","Data":"21d6cf7b406a6ff9cedd8ae6f8cd1cc4ac70cf199200ab7265de25b17187aeb7"} Oct 30 18:37:22.596387 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:22.596363 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446946k" Oct 30 18:37:22.755519 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:22.755445 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d5b7017-2445-45a3-aa04-ba33d1bb268e-util\" (UniqueName: \"kubernetes.io/empty-dir/5d5b7017-2445-45a3-aa04-ba33d1bb268e-util\") pod \"5d5b7017-2445-45a3-aa04-ba33d1bb268e\" (UID: \"5d5b7017-2445-45a3-aa04-ba33d1bb268e\") " Oct 30 18:37:22.755519 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:22.755477 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d5b7017-2445-45a3-aa04-ba33d1bb268e-kube-api-access-cnkks\" (UniqueName: \"kubernetes.io/projected/5d5b7017-2445-45a3-aa04-ba33d1bb268e-kube-api-access-cnkks\") pod \"5d5b7017-2445-45a3-aa04-ba33d1bb268e\" (UID: \"5d5b7017-2445-45a3-aa04-ba33d1bb268e\") " Oct 30 18:37:22.755731 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:22.755548 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d5b7017-2445-45a3-aa04-ba33d1bb268e-bundle\" (UniqueName: \"kubernetes.io/empty-dir/5d5b7017-2445-45a3-aa04-ba33d1bb268e-bundle\") pod \"5d5b7017-2445-45a3-aa04-ba33d1bb268e\" (UID: \"5d5b7017-2445-45a3-aa04-ba33d1bb268e\") " Oct 30 18:37:22.756003 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:22.755976 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d5b7017-2445-45a3-aa04-ba33d1bb268e-bundle" pod "5d5b7017-2445-45a3-aa04-ba33d1bb268e" (UID: "5d5b7017-2445-45a3-aa04-ba33d1bb268e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:22.758238 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:22.758212 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d5b7017-2445-45a3-aa04-ba33d1bb268e-kube-api-access-cnkks" pod "5d5b7017-2445-45a3-aa04-ba33d1bb268e" (UID: "5d5b7017-2445-45a3-aa04-ba33d1bb268e"). InnerVolumeSpecName "kube-api-access-cnkks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:37:22.760474 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:22.760450 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d5b7017-2445-45a3-aa04-ba33d1bb268e-util" pod "5d5b7017-2445-45a3-aa04-ba33d1bb268e" (UID: "5d5b7017-2445-45a3-aa04-ba33d1bb268e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:22.856592 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:22.856571 6960 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5d5b7017-2445-45a3-aa04-ba33d1bb268e-util\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:37:22.856690 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:22.856599 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnkks\" (UniqueName: \"kubernetes.io/projected/5d5b7017-2445-45a3-aa04-ba33d1bb268e-kube-api-access-cnkks\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:37:22.856690 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:22.856613 6960 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5d5b7017-2445-45a3-aa04-ba33d1bb268e-bundle\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:37:23.474408 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:23.474377 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446946k" event={"ID":"5d5b7017-2445-45a3-aa04-ba33d1bb268e","Type":"ContainerDied","Data":"021c35fc5be8ac59f6a960edbf4d6f2076f655c7248248785cb5bf3bb53fc20b"} Oct 30 18:37:23.474568 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:23.474424 6960 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="021c35fc5be8ac59f6a960edbf4d6f2076f655c7248248785cb5bf3bb53fc20b" Oct 30 18:37:23.474568 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:23.474451 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446946k" Oct 30 18:37:23.930979 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:37:23.930947 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 18:37:26.452369 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:26.452332 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:26.751457 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:26.751373 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:27.875869 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:27.875836 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-qvpr5"] Oct 30 18:37:27.876273 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:27.876165 6960 cpu_manager.go:414] "containerMap: removing container" podUID="5d5b7017-2445-45a3-aa04-ba33d1bb268e" containerName="pull" Oct 30 18:37:27.876273 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:27.876176 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d5b7017-2445-45a3-aa04-ba33d1bb268e" containerName="pull" Oct 30 18:37:27.876273 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:27.876193 6960 cpu_manager.go:414] "containerMap: removing container" podUID="5d5b7017-2445-45a3-aa04-ba33d1bb268e" containerName="util" Oct 30 18:37:27.876273 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:27.876199 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d5b7017-2445-45a3-aa04-ba33d1bb268e" containerName="util" Oct 30 18:37:27.876273 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:27.876210 6960 cpu_manager.go:414] "containerMap: removing container" podUID="5d5b7017-2445-45a3-aa04-ba33d1bb268e" containerName="extract" Oct 30 18:37:27.876273 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:27.876214 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d5b7017-2445-45a3-aa04-ba33d1bb268e" containerName="extract" Oct 30 18:37:27.876273 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:27.876274 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d5b7017-2445-45a3-aa04-ba33d1bb268e" containerName="extract" Oct 30 18:37:27.942293 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:27.942253 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-qvpr5"] Oct 30 18:37:27.942444 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:27.942367 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-qvpr5" Oct 30 18:37:27.951358 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:27.951337 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Oct 30 18:37:27.951576 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:27.951558 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-sd225\"" Oct 30 18:37:27.951625 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:27.951573 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Oct 30 18:37:28.099272 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:28.099242 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vt6dq\" (UniqueName: \"kubernetes.io/projected/fe73fac5-f4c6-4c24-8c47-983eba84a693-kube-api-access-vt6dq\") pod \"jobset-operator-674cd59dc5-qvpr5\" (UID: \"fe73fac5-f4c6-4c24-8c47-983eba84a693\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-qvpr5" Oct 30 18:37:28.099272 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:28.099273 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fe73fac5-f4c6-4c24-8c47-983eba84a693-tmp\") pod \"jobset-operator-674cd59dc5-qvpr5\" (UID: \"fe73fac5-f4c6-4c24-8c47-983eba84a693\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-qvpr5" Oct 30 18:37:28.200091 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:28.200023 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vt6dq\" (UniqueName: \"kubernetes.io/projected/fe73fac5-f4c6-4c24-8c47-983eba84a693-kube-api-access-vt6dq\") pod \"jobset-operator-674cd59dc5-qvpr5\" (UID: \"fe73fac5-f4c6-4c24-8c47-983eba84a693\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-qvpr5" Oct 30 18:37:28.200091 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:28.200060 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fe73fac5-f4c6-4c24-8c47-983eba84a693-tmp\") pod \"jobset-operator-674cd59dc5-qvpr5\" (UID: \"fe73fac5-f4c6-4c24-8c47-983eba84a693\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-qvpr5" Oct 30 18:37:28.200439 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:28.200418 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fe73fac5-f4c6-4c24-8c47-983eba84a693-tmp\") pod \"jobset-operator-674cd59dc5-qvpr5\" (UID: \"fe73fac5-f4c6-4c24-8c47-983eba84a693\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-qvpr5" Oct 30 18:37:28.213077 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:28.213052 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vt6dq\" (UniqueName: \"kubernetes.io/projected/fe73fac5-f4c6-4c24-8c47-983eba84a693-kube-api-access-vt6dq\") pod \"jobset-operator-674cd59dc5-qvpr5\" (UID: \"fe73fac5-f4c6-4c24-8c47-983eba84a693\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-qvpr5" Oct 30 18:37:28.281055 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:28.281028 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-qvpr5" Oct 30 18:37:28.595646 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:28.595619 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-qvpr5"] Oct 30 18:37:28.598027 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:37:28.598000 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfe73fac5_f4c6_4c24_8c47_983eba84a693.slice/crio-2d236448d093f784345c57d966336a43e28eecc66512717994cf2b74c2a81221 WatchSource:0}: Error finding container 2d236448d093f784345c57d966336a43e28eecc66512717994cf2b74c2a81221: Status 404 returned error can't find the container with id 2d236448d093f784345c57d966336a43e28eecc66512717994cf2b74c2a81221 Oct 30 18:37:28.700873 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:28.700843 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:28.808732 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:28.808705 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:29.498318 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:29.498283 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-qvpr5" event={"ID":"fe73fac5-f4c6-4c24-8c47-983eba84a693","Type":"ContainerStarted","Data":"2d236448d093f784345c57d966336a43e28eecc66512717994cf2b74c2a81221"} Oct 30 18:37:31.509007 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:31.508919 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-qvpr5" event={"ID":"fe73fac5-f4c6-4c24-8c47-983eba84a693","Type":"ContainerStarted","Data":"51c525e94e081e022de3c6e557b324283cdf7ec428c537bd3409ea91f0239a47"} Oct 30 18:37:31.533065 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:31.533014 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-qvpr5" podStartSLOduration=1.8760214990000001 podStartE2EDuration="4.532997356s" totalImagesPullingTime="2.656975857s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:37:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:37:31.531523467 +0000 UTC m=+219.060554604" watchObservedRunningTime="2026-10-30 18:37:31.532997356 +0000 UTC m=+219.062028497" Oct 30 18:37:31.617092 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:31.617059 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:31.751770 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:31.751735 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:37.268932 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:37.268898 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:39.456755 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:39.456722 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:39.714486 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:39.714413 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:41.591117 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:41.591087 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-747fbbfc6c-hpgwk"] Oct 30 18:37:41.593950 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:41.593928 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-747fbbfc6c-hpgwk" Oct 30 18:37:41.610424 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:41.610400 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/65527879-5a76-4efe-ba5c-2bda6f82aa51-metrics-cert\") pod \"opendatahub-operator-controller-manager-747fbbfc6c-hpgwk\" (UID: \"65527879-5a76-4efe-ba5c-2bda6f82aa51\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-747fbbfc6c-hpgwk" Oct 30 18:37:41.610544 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:41.610431 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lggvc\" (UniqueName: \"kubernetes.io/projected/65527879-5a76-4efe-ba5c-2bda6f82aa51-kube-api-access-lggvc\") pod \"opendatahub-operator-controller-manager-747fbbfc6c-hpgwk\" (UID: \"65527879-5a76-4efe-ba5c-2bda6f82aa51\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-747fbbfc6c-hpgwk" Oct 30 18:37:41.610544 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:41.610467 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65527879-5a76-4efe-ba5c-2bda6f82aa51-tmp\") pod \"opendatahub-operator-controller-manager-747fbbfc6c-hpgwk\" (UID: \"65527879-5a76-4efe-ba5c-2bda6f82aa51\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-747fbbfc6c-hpgwk" Oct 30 18:37:41.610628 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:41.610544 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/65527879-5a76-4efe-ba5c-2bda6f82aa51-apiservice-cert\") pod \"opendatahub-operator-controller-manager-747fbbfc6c-hpgwk\" (UID: \"65527879-5a76-4efe-ba5c-2bda6f82aa51\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-747fbbfc6c-hpgwk" Oct 30 18:37:41.610628 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:41.610579 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/65527879-5a76-4efe-ba5c-2bda6f82aa51-manifests\") pod \"opendatahub-operator-controller-manager-747fbbfc6c-hpgwk\" (UID: \"65527879-5a76-4efe-ba5c-2bda6f82aa51\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-747fbbfc6c-hpgwk" Oct 30 18:37:41.610628 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:41.610618 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/65527879-5a76-4efe-ba5c-2bda6f82aa51-webhook-cert\") pod \"opendatahub-operator-controller-manager-747fbbfc6c-hpgwk\" (UID: \"65527879-5a76-4efe-ba5c-2bda6f82aa51\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-747fbbfc6c-hpgwk" Oct 30 18:37:41.615240 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:41.615226 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 18:37:41.640333 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:41.640309 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 30 18:37:41.640621 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:41.640603 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 30 18:37:41.677507 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:41.677448 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-747fbbfc6c-hpgwk"] Oct 30 18:37:41.711321 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:41.711295 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/65527879-5a76-4efe-ba5c-2bda6f82aa51-metrics-cert\") pod \"opendatahub-operator-controller-manager-747fbbfc6c-hpgwk\" (UID: \"65527879-5a76-4efe-ba5c-2bda6f82aa51\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-747fbbfc6c-hpgwk" Oct 30 18:37:41.711450 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:41.711325 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lggvc\" (UniqueName: \"kubernetes.io/projected/65527879-5a76-4efe-ba5c-2bda6f82aa51-kube-api-access-lggvc\") pod \"opendatahub-operator-controller-manager-747fbbfc6c-hpgwk\" (UID: \"65527879-5a76-4efe-ba5c-2bda6f82aa51\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-747fbbfc6c-hpgwk" Oct 30 18:37:41.711450 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:41.711363 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65527879-5a76-4efe-ba5c-2bda6f82aa51-tmp\") pod \"opendatahub-operator-controller-manager-747fbbfc6c-hpgwk\" (UID: \"65527879-5a76-4efe-ba5c-2bda6f82aa51\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-747fbbfc6c-hpgwk" Oct 30 18:37:41.711520 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:41.711482 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/65527879-5a76-4efe-ba5c-2bda6f82aa51-apiservice-cert\") pod \"opendatahub-operator-controller-manager-747fbbfc6c-hpgwk\" (UID: \"65527879-5a76-4efe-ba5c-2bda6f82aa51\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-747fbbfc6c-hpgwk" Oct 30 18:37:41.711553 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:41.711526 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/65527879-5a76-4efe-ba5c-2bda6f82aa51-manifests\") pod \"opendatahub-operator-controller-manager-747fbbfc6c-hpgwk\" (UID: \"65527879-5a76-4efe-ba5c-2bda6f82aa51\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-747fbbfc6c-hpgwk" Oct 30 18:37:41.711610 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:41.711594 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/65527879-5a76-4efe-ba5c-2bda6f82aa51-webhook-cert\") pod \"opendatahub-operator-controller-manager-747fbbfc6c-hpgwk\" (UID: \"65527879-5a76-4efe-ba5c-2bda6f82aa51\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-747fbbfc6c-hpgwk" Oct 30 18:37:41.711840 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:41.711824 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/65527879-5a76-4efe-ba5c-2bda6f82aa51-manifests\") pod \"opendatahub-operator-controller-manager-747fbbfc6c-hpgwk\" (UID: \"65527879-5a76-4efe-ba5c-2bda6f82aa51\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-747fbbfc6c-hpgwk" Oct 30 18:37:41.713681 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:41.713661 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65527879-5a76-4efe-ba5c-2bda6f82aa51-tmp\") pod \"opendatahub-operator-controller-manager-747fbbfc6c-hpgwk\" (UID: \"65527879-5a76-4efe-ba5c-2bda6f82aa51\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-747fbbfc6c-hpgwk" Oct 30 18:37:41.717975 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:41.717958 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 18:37:41.718325 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:41.718306 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-c242x\"" Oct 30 18:37:41.718498 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:41.718483 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 30 18:37:41.723959 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:41.723937 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/65527879-5a76-4efe-ba5c-2bda6f82aa51-apiservice-cert\") pod \"opendatahub-operator-controller-manager-747fbbfc6c-hpgwk\" (UID: \"65527879-5a76-4efe-ba5c-2bda6f82aa51\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-747fbbfc6c-hpgwk" Oct 30 18:37:41.724043 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:41.723977 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/65527879-5a76-4efe-ba5c-2bda6f82aa51-webhook-cert\") pod \"opendatahub-operator-controller-manager-747fbbfc6c-hpgwk\" (UID: \"65527879-5a76-4efe-ba5c-2bda6f82aa51\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-747fbbfc6c-hpgwk" Oct 30 18:37:41.724043 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:41.723989 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/65527879-5a76-4efe-ba5c-2bda6f82aa51-metrics-cert\") pod \"opendatahub-operator-controller-manager-747fbbfc6c-hpgwk\" (UID: \"65527879-5a76-4efe-ba5c-2bda6f82aa51\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-747fbbfc6c-hpgwk" Oct 30 18:37:41.727909 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:41.727887 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lggvc\" (UniqueName: \"kubernetes.io/projected/65527879-5a76-4efe-ba5c-2bda6f82aa51-kube-api-access-lggvc\") pod \"opendatahub-operator-controller-manager-747fbbfc6c-hpgwk\" (UID: \"65527879-5a76-4efe-ba5c-2bda6f82aa51\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-747fbbfc6c-hpgwk" Oct 30 18:37:41.751041 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:41.751010 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:41.906633 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:41.906577 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-747fbbfc6c-hpgwk" Oct 30 18:37:42.287347 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:42.287268 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:42.301602 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:42.301465 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-747fbbfc6c-hpgwk"] Oct 30 18:37:42.301602 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:37:42.301567 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65527879_5a76_4efe_ba5c_2bda6f82aa51.slice/crio-c18a11e9f4766ee6cb94bdfc9595672865db22c44f1dffa6847dc4820f39cc86 WatchSource:0}: Error finding container c18a11e9f4766ee6cb94bdfc9595672865db22c44f1dffa6847dc4820f39cc86: Status 404 returned error can't find the container with id c18a11e9f4766ee6cb94bdfc9595672865db22c44f1dffa6847dc4820f39cc86 Oct 30 18:37:42.560730 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:42.560698 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-747fbbfc6c-hpgwk" event={"ID":"65527879-5a76-4efe-ba5c-2bda6f82aa51","Type":"ContainerStarted","Data":"c18a11e9f4766ee6cb94bdfc9595672865db22c44f1dffa6847dc4820f39cc86"} Oct 30 18:37:42.700857 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:42.700819 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:43.362487 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:43.362392 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:43.838180 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:43.838108 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:45.114189 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:45.114064 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:45.388218 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:45.388137 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:45.577312 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:45.577279 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-747fbbfc6c-hpgwk" event={"ID":"65527879-5a76-4efe-ba5c-2bda6f82aa51","Type":"ContainerStarted","Data":"a332db2b6efee827facb137b29c9446a2dc0894c38501b8367b6422f3c74db99"} Oct 30 18:37:46.583828 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:46.583794 6960 generic.go:356] "Generic (PLEG): container finished" podID="65527879-5a76-4efe-ba5c-2bda6f82aa51" containerID="a332db2b6efee827facb137b29c9446a2dc0894c38501b8367b6422f3c74db99" exitCode=0 Oct 30 18:37:46.584249 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:46.583884 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-747fbbfc6c-hpgwk" event={"ID":"65527879-5a76-4efe-ba5c-2bda6f82aa51","Type":"ContainerDied","Data":"a332db2b6efee827facb137b29c9446a2dc0894c38501b8367b6422f3c74db99"} Oct 30 18:37:46.625000 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:46.624971 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:46.992827 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:46.992801 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:47.586442 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:47.586400 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:47.590267 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:47.590239 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-747fbbfc6c-hpgwk" event={"ID":"65527879-5a76-4efe-ba5c-2bda6f82aa51","Type":"ContainerStarted","Data":"8a3e0d68faeae5b40c90b0965f42729c5b7554876d7b1e81c8de78d68c4aa97f"} Oct 30 18:37:47.590414 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:47.590355 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-747fbbfc6c-hpgwk" Oct 30 18:37:47.827702 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:47.827648 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-747fbbfc6c-hpgwk" podStartSLOduration=3.050738592 podStartE2EDuration="6.827635603s" totalImagesPullingTime="3.310426594s" totalInitContainerRuntime="466.470417ms" isStatefulPod=false podCreationTimestamp="2026-10-30 18:37:41 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:37:47.810735772 +0000 UTC m=+235.339766910" watchObservedRunningTime="2026-10-30 18:37:47.827635603 +0000 UTC m=+235.356666740" Oct 30 18:37:48.254060 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:48.254029 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:49.278944 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:49.278907 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:50.041549 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:50.041516 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:50.411565 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:50.411533 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:50.927701 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:50.927659 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:51.370622 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:51.370592 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:53.930786 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:37:53.930738 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 18:37:54.052259 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:54.052225 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:54.824372 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:54.824330 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:57.379724 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:57.379683 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-rmz2c"] Oct 30 18:37:57.384764 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:57.384740 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-rmz2c" Oct 30 18:37:57.399646 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:57.399627 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Oct 30 18:37:57.399797 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:57.399780 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Oct 30 18:37:57.400903 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:57.400888 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-cz47w\"" Oct 30 18:37:57.411134 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:57.411112 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-rmz2c"] Oct 30 18:37:57.454312 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:57.454288 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3e445b83-3ba7-4967-bd04-8b9598072d25-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-rmz2c\" (UID: \"3e445b83-3ba7-4967-bd04-8b9598072d25\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-rmz2c" Oct 30 18:37:57.454445 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:57.454428 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwxkk\" (UniqueName: \"kubernetes.io/projected/3e445b83-3ba7-4967-bd04-8b9598072d25-kube-api-access-cwxkk\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-rmz2c\" (UID: \"3e445b83-3ba7-4967-bd04-8b9598072d25\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-rmz2c" Oct 30 18:37:57.555190 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:57.555163 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3e445b83-3ba7-4967-bd04-8b9598072d25-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-rmz2c\" (UID: \"3e445b83-3ba7-4967-bd04-8b9598072d25\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-rmz2c" Oct 30 18:37:57.555323 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:57.555295 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwxkk\" (UniqueName: \"kubernetes.io/projected/3e445b83-3ba7-4967-bd04-8b9598072d25-kube-api-access-cwxkk\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-rmz2c\" (UID: \"3e445b83-3ba7-4967-bd04-8b9598072d25\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-rmz2c" Oct 30 18:37:57.555437 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:57.555416 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3e445b83-3ba7-4967-bd04-8b9598072d25-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-rmz2c\" (UID: \"3e445b83-3ba7-4967-bd04-8b9598072d25\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-rmz2c" Oct 30 18:37:57.570275 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:57.570252 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwxkk\" (UniqueName: \"kubernetes.io/projected/3e445b83-3ba7-4967-bd04-8b9598072d25-kube-api-access-cwxkk\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-rmz2c\" (UID: \"3e445b83-3ba7-4967-bd04-8b9598072d25\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-rmz2c" Oct 30 18:37:57.695648 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:57.695580 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-rmz2c" Oct 30 18:37:57.987915 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:57.987892 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-rmz2c"] Oct 30 18:37:57.990407 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:37:57.990380 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e445b83_3ba7_4967_bd04_8b9598072d25.slice/crio-487e3ea9b32741dc39b61d48a975731be6ea9db79db60c1acb2d56283675049c WatchSource:0}: Error finding container 487e3ea9b32741dc39b61d48a975731be6ea9db79db60c1acb2d56283675049c: Status 404 returned error can't find the container with id 487e3ea9b32741dc39b61d48a975731be6ea9db79db60c1acb2d56283675049c Oct 30 18:37:58.086612 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:58.086584 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:58.596340 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:58.596315 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-747fbbfc6c-hpgwk" Oct 30 18:37:58.632989 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:37:58.632960 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-rmz2c" event={"ID":"3e445b83-3ba7-4967-bd04-8b9598072d25","Type":"ContainerStarted","Data":"487e3ea9b32741dc39b61d48a975731be6ea9db79db60c1acb2d56283675049c"} Oct 30 18:38:00.013567 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:00.013533 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-hm48j"] Oct 30 18:38:00.021276 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:00.021252 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-hm48j" Oct 30 18:38:00.038664 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:00.038632 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 18:38:00.038846 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:00.038814 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 18:38:00.038980 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:00.038862 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-tclbv\"" Oct 30 18:38:00.052770 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:00.052740 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-hm48j"] Oct 30 18:38:00.094738 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:00.094703 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54n67\" (UniqueName: \"kubernetes.io/projected/a624d24b-f689-4286-9602-c89d828a0e05-kube-api-access-54n67\") pod \"authorino-operator-7b7bb4bc77-hm48j\" (UID: \"a624d24b-f689-4286-9602-c89d828a0e05\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-hm48j" Oct 30 18:38:00.195525 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:00.195494 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54n67\" (UniqueName: \"kubernetes.io/projected/a624d24b-f689-4286-9602-c89d828a0e05-kube-api-access-54n67\") pod \"authorino-operator-7b7bb4bc77-hm48j\" (UID: \"a624d24b-f689-4286-9602-c89d828a0e05\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-hm48j" Oct 30 18:38:00.210730 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:00.210702 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54n67\" (UniqueName: \"kubernetes.io/projected/a624d24b-f689-4286-9602-c89d828a0e05-kube-api-access-54n67\") pod \"authorino-operator-7b7bb4bc77-hm48j\" (UID: \"a624d24b-f689-4286-9602-c89d828a0e05\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-hm48j" Oct 30 18:38:00.337299 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:00.337271 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-hm48j" Oct 30 18:38:00.650335 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:00.650278 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-hm48j"] Oct 30 18:38:00.653054 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:38:00.653028 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda624d24b_f689_4286_9602_c89d828a0e05.slice/crio-f75911323ad9be9d9025a92a1891e8bf7c4a47e2f913ac1f235f678a22c6f733 WatchSource:0}: Error finding container f75911323ad9be9d9025a92a1891e8bf7c4a47e2f913ac1f235f678a22c6f733: Status 404 returned error can't find the container with id f75911323ad9be9d9025a92a1891e8bf7c4a47e2f913ac1f235f678a22c6f733 Oct 30 18:38:00.680349 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:00.680321 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:00.939216 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:00.939121 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:01.552015 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:01.551985 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-f78tf"] Oct 30 18:38:01.556344 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:01.556311 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-f78tf" Oct 30 18:38:01.583045 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:01.583019 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Oct 30 18:38:01.583227 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:01.583087 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-hdz2z\"" Oct 30 18:38:01.595538 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:01.595511 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:01.609468 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:01.609440 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Oct 30 18:38:01.648369 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:01.648341 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-rmz2c" event={"ID":"3e445b83-3ba7-4967-bd04-8b9598072d25","Type":"ContainerStarted","Data":"2908a8004180b64591664e0f5d6b60e5a4fc8f74f2af0232b82a7be223a91b7b"} Oct 30 18:38:01.650201 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:01.650172 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-hm48j" event={"ID":"a624d24b-f689-4286-9602-c89d828a0e05","Type":"ContainerStarted","Data":"f75911323ad9be9d9025a92a1891e8bf7c4a47e2f913ac1f235f678a22c6f733"} Oct 30 18:38:01.663700 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:01.663665 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-f78tf"] Oct 30 18:38:01.709709 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:01.709578 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/30c4e293-73c6-4d1f-8e76-6ffb15e03f51-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-f78tf\" (UID: \"30c4e293-73c6-4d1f-8e76-6ffb15e03f51\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-f78tf" Oct 30 18:38:01.709709 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:01.709623 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52qrq\" (UniqueName: \"kubernetes.io/projected/30c4e293-73c6-4d1f-8e76-6ffb15e03f51-kube-api-access-52qrq\") pod \"kuadrant-console-plugin-66c4fb575d-f78tf\" (UID: \"30c4e293-73c6-4d1f-8e76-6ffb15e03f51\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-f78tf" Oct 30 18:38:01.709709 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:01.709663 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/30c4e293-73c6-4d1f-8e76-6ffb15e03f51-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-f78tf\" (UID: \"30c4e293-73c6-4d1f-8e76-6ffb15e03f51\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-f78tf" Oct 30 18:38:01.811232 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:01.811188 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/30c4e293-73c6-4d1f-8e76-6ffb15e03f51-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-f78tf\" (UID: \"30c4e293-73c6-4d1f-8e76-6ffb15e03f51\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-f78tf" Oct 30 18:38:01.811422 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:01.811247 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52qrq\" (UniqueName: \"kubernetes.io/projected/30c4e293-73c6-4d1f-8e76-6ffb15e03f51-kube-api-access-52qrq\") pod \"kuadrant-console-plugin-66c4fb575d-f78tf\" (UID: \"30c4e293-73c6-4d1f-8e76-6ffb15e03f51\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-f78tf" Oct 30 18:38:01.812128 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:01.811545 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/30c4e293-73c6-4d1f-8e76-6ffb15e03f51-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-f78tf\" (UID: \"30c4e293-73c6-4d1f-8e76-6ffb15e03f51\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-f78tf" Oct 30 18:38:01.812128 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:01.812080 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/30c4e293-73c6-4d1f-8e76-6ffb15e03f51-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-f78tf\" (UID: \"30c4e293-73c6-4d1f-8e76-6ffb15e03f51\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-f78tf" Oct 30 18:38:01.814441 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:01.814418 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/30c4e293-73c6-4d1f-8e76-6ffb15e03f51-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-f78tf\" (UID: \"30c4e293-73c6-4d1f-8e76-6ffb15e03f51\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-f78tf" Oct 30 18:38:01.838895 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:01.838848 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52qrq\" (UniqueName: \"kubernetes.io/projected/30c4e293-73c6-4d1f-8e76-6ffb15e03f51-kube-api-access-52qrq\") pod \"kuadrant-console-plugin-66c4fb575d-f78tf\" (UID: \"30c4e293-73c6-4d1f-8e76-6ffb15e03f51\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-f78tf" Oct 30 18:38:01.871472 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:01.871442 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-f78tf" Oct 30 18:38:01.964064 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:01.964032 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:02.211227 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:02.210571 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-rmz2c" podStartSLOduration=2.562421866 podStartE2EDuration="5.210553922s" totalImagesPullingTime="2.648132056s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:37:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:01.706907888 +0000 UTC m=+249.235939025" watchObservedRunningTime="2026-10-30 18:38:02.210553922 +0000 UTC m=+249.739585059" Oct 30 18:38:02.212554 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:02.212532 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-f78tf"] Oct 30 18:38:02.215429 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:38:02.215405 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod30c4e293_73c6_4d1f_8e76_6ffb15e03f51.slice/crio-afc26a4989c77753f5ed249fc1f171ac0a019e613071813de41e7f71a91532bf WatchSource:0}: Error finding container afc26a4989c77753f5ed249fc1f171ac0a019e613071813de41e7f71a91532bf: Status 404 returned error can't find the container with id afc26a4989c77753f5ed249fc1f171ac0a019e613071813de41e7f71a91532bf Oct 30 18:38:02.656611 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:02.656562 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-f78tf" event={"ID":"30c4e293-73c6-4d1f-8e76-6ffb15e03f51","Type":"ContainerStarted","Data":"afc26a4989c77753f5ed249fc1f171ac0a019e613071813de41e7f71a91532bf"} Oct 30 18:38:03.112066 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:03.111957 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:03.447321 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:03.447241 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:03.467100 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:03.467072 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-8fq69"] Oct 30 18:38:03.470904 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:03.470880 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8fq69" Oct 30 18:38:03.477364 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:03.475822 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 30 18:38:03.481590 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:03.481499 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 30 18:38:03.502412 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:03.502378 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-b5l6d\"" Oct 30 18:38:03.511022 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:03.507299 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-8fq69"] Oct 30 18:38:03.527673 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:03.527651 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/557f8cec-b11d-4faf-8d51-ca780d9381ca-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-8fq69\" (UID: \"557f8cec-b11d-4faf-8d51-ca780d9381ca\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8fq69" Oct 30 18:38:03.528805 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:03.527810 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bg2t\" (UniqueName: \"kubernetes.io/projected/557f8cec-b11d-4faf-8d51-ca780d9381ca-kube-api-access-4bg2t\") pod \"cert-manager-webhook-5dbbbf75cc-8fq69\" (UID: \"557f8cec-b11d-4faf-8d51-ca780d9381ca\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8fq69" Oct 30 18:38:03.628332 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:03.628304 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/557f8cec-b11d-4faf-8d51-ca780d9381ca-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-8fq69\" (UID: \"557f8cec-b11d-4faf-8d51-ca780d9381ca\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8fq69" Oct 30 18:38:03.628485 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:03.628375 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bg2t\" (UniqueName: \"kubernetes.io/projected/557f8cec-b11d-4faf-8d51-ca780d9381ca-kube-api-access-4bg2t\") pod \"cert-manager-webhook-5dbbbf75cc-8fq69\" (UID: \"557f8cec-b11d-4faf-8d51-ca780d9381ca\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8fq69" Oct 30 18:38:03.660296 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:03.660266 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bg2t\" (UniqueName: \"kubernetes.io/projected/557f8cec-b11d-4faf-8d51-ca780d9381ca-kube-api-access-4bg2t\") pod \"cert-manager-webhook-5dbbbf75cc-8fq69\" (UID: \"557f8cec-b11d-4faf-8d51-ca780d9381ca\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8fq69" Oct 30 18:38:03.663609 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:03.663572 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-hm48j" event={"ID":"a624d24b-f689-4286-9602-c89d828a0e05","Type":"ContainerStarted","Data":"24b64c01c7b93d202e79680d39f51e8ef3cbccb2f2f617aaa2b7101b22839f2c"} Oct 30 18:38:03.663902 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:03.663844 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-hm48j" Oct 30 18:38:03.673747 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:03.673717 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/557f8cec-b11d-4faf-8d51-ca780d9381ca-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-8fq69\" (UID: \"557f8cec-b11d-4faf-8d51-ca780d9381ca\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8fq69" Oct 30 18:38:03.705305 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:03.705219 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-hm48j" podStartSLOduration=1.843166617 podStartE2EDuration="4.705203555s" totalImagesPullingTime="2.862036938s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:37:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:03.703853785 +0000 UTC m=+251.232884923" watchObservedRunningTime="2026-10-30 18:38:03.705203555 +0000 UTC m=+251.234234695" Oct 30 18:38:03.788541 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:03.788513 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8fq69" Oct 30 18:38:04.098154 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:04.098112 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-8fq69"] Oct 30 18:38:04.098422 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:38:04.098393 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod557f8cec_b11d_4faf_8d51_ca780d9381ca.slice/crio-765e4552582a0250a3367cbd80a37ffd28e58203a5bba2afcfd16799a10965eb WatchSource:0}: Error finding container 765e4552582a0250a3367cbd80a37ffd28e58203a5bba2afcfd16799a10965eb: Status 404 returned error can't find the container with id 765e4552582a0250a3367cbd80a37ffd28e58203a5bba2afcfd16799a10965eb Oct 30 18:38:04.147347 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:04.147315 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:04.494801 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:04.494698 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:04.669593 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:04.669558 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8fq69" event={"ID":"557f8cec-b11d-4faf-8d51-ca780d9381ca","Type":"ContainerStarted","Data":"765e4552582a0250a3367cbd80a37ffd28e58203a5bba2afcfd16799a10965eb"} Oct 30 18:38:05.666888 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:05.666832 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:06.131339 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:06.131305 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:07.686769 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:07.686681 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-f78tf" event={"ID":"30c4e293-73c6-4d1f-8e76-6ffb15e03f51","Type":"ContainerStarted","Data":"3be4fa3171747085cf12747ae84ae0957d04410d5b9f1594f217f48702c89e6a"} Oct 30 18:38:07.688399 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:07.688368 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8fq69" event={"ID":"557f8cec-b11d-4faf-8d51-ca780d9381ca","Type":"ContainerStarted","Data":"ee07863dde95d613ee5336ff3d7ddf5ce14ccdbf1b0d80ce1a9720b6a9d6ccc8"} Oct 30 18:38:07.688508 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:07.688432 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8fq69" Oct 30 18:38:07.751766 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:07.751722 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-f78tf" podStartSLOduration=1.617853199 podStartE2EDuration="6.751708559s" totalImagesPullingTime="5.13385536s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:38:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:07.750421985 +0000 UTC m=+255.279453123" watchObservedRunningTime="2026-10-30 18:38:07.751708559 +0000 UTC m=+255.280739696" Oct 30 18:38:07.797664 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:07.797619 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8fq69" podStartSLOduration=1.498739918 podStartE2EDuration="4.797605099s" totalImagesPullingTime="3.298865181s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:38:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:07.796120456 +0000 UTC m=+255.325151595" watchObservedRunningTime="2026-10-30 18:38:07.797605099 +0000 UTC m=+255.326636236" Oct 30 18:38:08.727963 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:08.727926 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:09.058423 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:09.058380 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:11.236122 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:11.236079 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:11.482512 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:11.482480 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:12.224464 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:12.224426 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:12.637403 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:12.637365 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:12.728897 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:12.728864 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-65c8c79996-jjcwx"] Oct 30 18:38:12.733552 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:12.733533 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-65c8c79996-jjcwx" Oct 30 18:38:12.809889 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:12.809856 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 30 18:38:12.810039 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:12.809915 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 30 18:38:12.810039 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:12.809962 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-9s2fw\"" Oct 30 18:38:12.812400 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:12.812380 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fa35d05a-b5c7-40b2-ac7a-6d72ebf85a69-tmp\") pod \"kube-auth-proxy-65c8c79996-jjcwx\" (UID: \"fa35d05a-b5c7-40b2-ac7a-6d72ebf85a69\") " pod="openshift-ingress/kube-auth-proxy-65c8c79996-jjcwx" Oct 30 18:38:12.812490 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:12.812439 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa35d05a-b5c7-40b2-ac7a-6d72ebf85a69-tls-certs\") pod \"kube-auth-proxy-65c8c79996-jjcwx\" (UID: \"fa35d05a-b5c7-40b2-ac7a-6d72ebf85a69\") " pod="openshift-ingress/kube-auth-proxy-65c8c79996-jjcwx" Oct 30 18:38:12.812583 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:12.812565 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rlxq\" (UniqueName: \"kubernetes.io/projected/fa35d05a-b5c7-40b2-ac7a-6d72ebf85a69-kube-api-access-6rlxq\") pod \"kube-auth-proxy-65c8c79996-jjcwx\" (UID: \"fa35d05a-b5c7-40b2-ac7a-6d72ebf85a69\") " pod="openshift-ingress/kube-auth-proxy-65c8c79996-jjcwx" Oct 30 18:38:12.816757 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:12.816737 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-65c8c79996-jjcwx"] Oct 30 18:38:12.913550 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:12.913481 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fa35d05a-b5c7-40b2-ac7a-6d72ebf85a69-tmp\") pod \"kube-auth-proxy-65c8c79996-jjcwx\" (UID: \"fa35d05a-b5c7-40b2-ac7a-6d72ebf85a69\") " pod="openshift-ingress/kube-auth-proxy-65c8c79996-jjcwx" Oct 30 18:38:12.913550 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:12.913537 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa35d05a-b5c7-40b2-ac7a-6d72ebf85a69-tls-certs\") pod \"kube-auth-proxy-65c8c79996-jjcwx\" (UID: \"fa35d05a-b5c7-40b2-ac7a-6d72ebf85a69\") " pod="openshift-ingress/kube-auth-proxy-65c8c79996-jjcwx" Oct 30 18:38:12.913719 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:38:12.913646 6960 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Oct 30 18:38:12.913755 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:38:12.913718 6960 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa35d05a-b5c7-40b2-ac7a-6d72ebf85a69-tls-certs podName:fa35d05a-b5c7-40b2-ac7a-6d72ebf85a69 nodeName:}" failed. No retries permitted until 2026-10-30 18:38:13.413696176 +0000 UTC m=+260.942727310 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/fa35d05a-b5c7-40b2-ac7a-6d72ebf85a69-tls-certs") pod "kube-auth-proxy-65c8c79996-jjcwx" (UID: "fa35d05a-b5c7-40b2-ac7a-6d72ebf85a69") : secret "kube-auth-proxy-tls" not found Oct 30 18:38:12.913755 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:12.913733 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rlxq\" (UniqueName: \"kubernetes.io/projected/fa35d05a-b5c7-40b2-ac7a-6d72ebf85a69-kube-api-access-6rlxq\") pod \"kube-auth-proxy-65c8c79996-jjcwx\" (UID: \"fa35d05a-b5c7-40b2-ac7a-6d72ebf85a69\") " pod="openshift-ingress/kube-auth-proxy-65c8c79996-jjcwx" Oct 30 18:38:12.915754 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:12.915730 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fa35d05a-b5c7-40b2-ac7a-6d72ebf85a69-tmp\") pod \"kube-auth-proxy-65c8c79996-jjcwx\" (UID: \"fa35d05a-b5c7-40b2-ac7a-6d72ebf85a69\") " pod="openshift-ingress/kube-auth-proxy-65c8c79996-jjcwx" Oct 30 18:38:12.964030 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:12.963981 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rlxq\" (UniqueName: \"kubernetes.io/projected/fa35d05a-b5c7-40b2-ac7a-6d72ebf85a69-kube-api-access-6rlxq\") pod \"kube-auth-proxy-65c8c79996-jjcwx\" (UID: \"fa35d05a-b5c7-40b2-ac7a-6d72ebf85a69\") " pod="openshift-ingress/kube-auth-proxy-65c8c79996-jjcwx" Oct 30 18:38:13.054103 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:13.054075 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c55722t75"] Oct 30 18:38:13.057699 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:13.057681 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c55722t75" Oct 30 18:38:13.080094 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:13.080075 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-zpzt9\"" Oct 30 18:38:13.098324 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:13.098300 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c55722t75"] Oct 30 18:38:13.171810 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:13.171734 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:13.215998 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:13.215968 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b0ef8f8-8b08-40e1-8338-fd48c63cfcf7-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68d996c55722t75\" (UID: \"9b0ef8f8-8b08-40e1-8338-fd48c63cfcf7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c55722t75" Oct 30 18:38:13.216196 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:13.216013 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b0ef8f8-8b08-40e1-8338-fd48c63cfcf7-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68d996c55722t75\" (UID: \"9b0ef8f8-8b08-40e1-8338-fd48c63cfcf7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c55722t75" Oct 30 18:38:13.216196 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:13.216031 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b0ef8f8-8b08-40e1-8338-fd48c63cfcf7-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68d996c55722t75\" (UID: \"9b0ef8f8-8b08-40e1-8338-fd48c63cfcf7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c55722t75" Oct 30 18:38:13.216196 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:13.216121 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b0ef8f8-8b08-40e1-8338-fd48c63cfcf7-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68d996c55722t75\" (UID: \"9b0ef8f8-8b08-40e1-8338-fd48c63cfcf7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c55722t75" Oct 30 18:38:13.216369 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:13.216247 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b0ef8f8-8b08-40e1-8338-fd48c63cfcf7-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68d996c55722t75\" (UID: \"9b0ef8f8-8b08-40e1-8338-fd48c63cfcf7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c55722t75" Oct 30 18:38:13.216369 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:13.216288 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b0ef8f8-8b08-40e1-8338-fd48c63cfcf7-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68d996c55722t75\" (UID: \"9b0ef8f8-8b08-40e1-8338-fd48c63cfcf7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c55722t75" Oct 30 18:38:13.216369 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:13.216305 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b0ef8f8-8b08-40e1-8338-fd48c63cfcf7-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68d996c55722t75\" (UID: \"9b0ef8f8-8b08-40e1-8338-fd48c63cfcf7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c55722t75" Oct 30 18:38:13.216526 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:13.216384 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b0ef8f8-8b08-40e1-8338-fd48c63cfcf7-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68d996c55722t75\" (UID: \"9b0ef8f8-8b08-40e1-8338-fd48c63cfcf7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c55722t75" Oct 30 18:38:13.216526 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:13.216411 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68tl7\" (UniqueName: \"kubernetes.io/projected/9b0ef8f8-8b08-40e1-8338-fd48c63cfcf7-kube-api-access-68tl7\") pod \"data-science-gateway-data-science-gateway-class-68d996c55722t75\" (UID: \"9b0ef8f8-8b08-40e1-8338-fd48c63cfcf7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c55722t75" Oct 30 18:38:13.317715 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:13.317680 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b0ef8f8-8b08-40e1-8338-fd48c63cfcf7-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68d996c55722t75\" (UID: \"9b0ef8f8-8b08-40e1-8338-fd48c63cfcf7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c55722t75" Oct 30 18:38:13.317715 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:13.317717 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b0ef8f8-8b08-40e1-8338-fd48c63cfcf7-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68d996c55722t75\" (UID: \"9b0ef8f8-8b08-40e1-8338-fd48c63cfcf7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c55722t75" Oct 30 18:38:13.317942 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:13.317745 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b0ef8f8-8b08-40e1-8338-fd48c63cfcf7-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68d996c55722t75\" (UID: \"9b0ef8f8-8b08-40e1-8338-fd48c63cfcf7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c55722t75" Oct 30 18:38:13.317942 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:13.317768 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68tl7\" (UniqueName: \"kubernetes.io/projected/9b0ef8f8-8b08-40e1-8338-fd48c63cfcf7-kube-api-access-68tl7\") pod \"data-science-gateway-data-science-gateway-class-68d996c55722t75\" (UID: \"9b0ef8f8-8b08-40e1-8338-fd48c63cfcf7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c55722t75" Oct 30 18:38:13.317942 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:13.317827 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b0ef8f8-8b08-40e1-8338-fd48c63cfcf7-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68d996c55722t75\" (UID: \"9b0ef8f8-8b08-40e1-8338-fd48c63cfcf7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c55722t75" Oct 30 18:38:13.317942 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:13.317888 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b0ef8f8-8b08-40e1-8338-fd48c63cfcf7-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68d996c55722t75\" (UID: \"9b0ef8f8-8b08-40e1-8338-fd48c63cfcf7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c55722t75" Oct 30 18:38:13.318174 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:13.317943 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b0ef8f8-8b08-40e1-8338-fd48c63cfcf7-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68d996c55722t75\" (UID: \"9b0ef8f8-8b08-40e1-8338-fd48c63cfcf7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c55722t75" Oct 30 18:38:13.318174 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:13.318012 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b0ef8f8-8b08-40e1-8338-fd48c63cfcf7-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68d996c55722t75\" (UID: \"9b0ef8f8-8b08-40e1-8338-fd48c63cfcf7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c55722t75" Oct 30 18:38:13.318174 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:13.318091 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b0ef8f8-8b08-40e1-8338-fd48c63cfcf7-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68d996c55722t75\" (UID: \"9b0ef8f8-8b08-40e1-8338-fd48c63cfcf7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c55722t75" Oct 30 18:38:13.318336 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:13.318202 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b0ef8f8-8b08-40e1-8338-fd48c63cfcf7-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68d996c55722t75\" (UID: \"9b0ef8f8-8b08-40e1-8338-fd48c63cfcf7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c55722t75" Oct 30 18:38:13.318336 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:13.318267 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b0ef8f8-8b08-40e1-8338-fd48c63cfcf7-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68d996c55722t75\" (UID: \"9b0ef8f8-8b08-40e1-8338-fd48c63cfcf7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c55722t75" Oct 30 18:38:13.318446 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:13.318424 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b0ef8f8-8b08-40e1-8338-fd48c63cfcf7-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68d996c55722t75\" (UID: \"9b0ef8f8-8b08-40e1-8338-fd48c63cfcf7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c55722t75" Oct 30 18:38:13.318523 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:13.318497 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b0ef8f8-8b08-40e1-8338-fd48c63cfcf7-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68d996c55722t75\" (UID: \"9b0ef8f8-8b08-40e1-8338-fd48c63cfcf7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c55722t75" Oct 30 18:38:13.318621 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:13.318599 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b0ef8f8-8b08-40e1-8338-fd48c63cfcf7-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68d996c55722t75\" (UID: \"9b0ef8f8-8b08-40e1-8338-fd48c63cfcf7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c55722t75" Oct 30 18:38:13.320332 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:13.320312 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b0ef8f8-8b08-40e1-8338-fd48c63cfcf7-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68d996c55722t75\" (UID: \"9b0ef8f8-8b08-40e1-8338-fd48c63cfcf7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c55722t75" Oct 30 18:38:13.320587 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:13.320568 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b0ef8f8-8b08-40e1-8338-fd48c63cfcf7-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68d996c55722t75\" (UID: \"9b0ef8f8-8b08-40e1-8338-fd48c63cfcf7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c55722t75" Oct 30 18:38:13.385514 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:13.385479 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68tl7\" (UniqueName: \"kubernetes.io/projected/9b0ef8f8-8b08-40e1-8338-fd48c63cfcf7-kube-api-access-68tl7\") pod \"data-science-gateway-data-science-gateway-class-68d996c55722t75\" (UID: \"9b0ef8f8-8b08-40e1-8338-fd48c63cfcf7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c55722t75" Oct 30 18:38:13.386322 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:13.386301 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b0ef8f8-8b08-40e1-8338-fd48c63cfcf7-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68d996c55722t75\" (UID: \"9b0ef8f8-8b08-40e1-8338-fd48c63cfcf7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c55722t75" Oct 30 18:38:13.419533 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:13.419506 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa35d05a-b5c7-40b2-ac7a-6d72ebf85a69-tls-certs\") pod \"kube-auth-proxy-65c8c79996-jjcwx\" (UID: \"fa35d05a-b5c7-40b2-ac7a-6d72ebf85a69\") " pod="openshift-ingress/kube-auth-proxy-65c8c79996-jjcwx" Oct 30 18:38:13.422000 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:13.421956 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa35d05a-b5c7-40b2-ac7a-6d72ebf85a69-tls-certs\") pod \"kube-auth-proxy-65c8c79996-jjcwx\" (UID: \"fa35d05a-b5c7-40b2-ac7a-6d72ebf85a69\") " pod="openshift-ingress/kube-auth-proxy-65c8c79996-jjcwx" Oct 30 18:38:13.645666 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:13.645636 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-65c8c79996-jjcwx" Oct 30 18:38:13.669861 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:13.669835 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c55722t75" Oct 30 18:38:13.678774 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:13.678731 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:13.697671 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:13.695857 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8fq69" Oct 30 18:38:13.949469 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:13.949446 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-65c8c79996-jjcwx"] Oct 30 18:38:13.951219 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:38:13.951189 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa35d05a_b5c7_40b2_ac7a_6d72ebf85a69.slice/crio-608cf924de112aa3ae116a9478406cb5354d09f6652aa3ee31970ddc333070e0 WatchSource:0}: Error finding container 608cf924de112aa3ae116a9478406cb5354d09f6652aa3ee31970ddc333070e0: Status 404 returned error can't find the container with id 608cf924de112aa3ae116a9478406cb5354d09f6652aa3ee31970ddc333070e0 Oct 30 18:38:13.972846 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:13.972819 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c55722t75"] Oct 30 18:38:13.975308 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:38:13.975283 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b0ef8f8_8b08_40e1_8338_fd48c63cfcf7.slice/crio-eac9d288dbb4996d2642f25984061faffa4c5c6cc5b1bdfd263825ae8e25b20e WatchSource:0}: Error finding container eac9d288dbb4996d2642f25984061faffa4c5c6cc5b1bdfd263825ae8e25b20e: Status 404 returned error can't find the container with id eac9d288dbb4996d2642f25984061faffa4c5c6cc5b1bdfd263825ae8e25b20e Oct 30 18:38:13.977565 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:13.977529 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:38:13.977649 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:13.977615 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:38:13.977686 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:13.977660 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:38:14.672663 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:14.671745 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-hm48j" Oct 30 18:38:14.724132 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:14.724047 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c55722t75" event={"ID":"9b0ef8f8-8b08-40e1-8338-fd48c63cfcf7","Type":"ContainerStarted","Data":"ee123ede37ee6163fd06d9a67be3c76ffb9ddca8cad1bb73544de2e25d045a17"} Oct 30 18:38:14.724132 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:14.724105 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c55722t75" event={"ID":"9b0ef8f8-8b08-40e1-8338-fd48c63cfcf7","Type":"ContainerStarted","Data":"eac9d288dbb4996d2642f25984061faffa4c5c6cc5b1bdfd263825ae8e25b20e"} Oct 30 18:38:14.725897 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:14.725862 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-65c8c79996-jjcwx" event={"ID":"fa35d05a-b5c7-40b2-ac7a-6d72ebf85a69","Type":"ContainerStarted","Data":"608cf924de112aa3ae116a9478406cb5354d09f6652aa3ee31970ddc333070e0"} Oct 30 18:38:14.869449 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:14.869324 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c55722t75" podStartSLOduration=2.8693011779999997 podStartE2EDuration="2.869301178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:38:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:14.86509014 +0000 UTC m=+262.394121306" watchObservedRunningTime="2026-10-30 18:38:14.869301178 +0000 UTC m=+262.398332320" Oct 30 18:38:14.871446 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:14.871406 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:15.517622 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:15.517589 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:15.670606 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:15.670569 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c55722t75" Oct 30 18:38:15.676160 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:15.676122 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c55722t75" Oct 30 18:38:15.731272 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:15.731246 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c55722t75" Oct 30 18:38:15.732347 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:15.732310 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68d996c55722t75" Oct 30 18:38:15.908862 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:15.908816 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:16.737460 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:16.737424 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-65c8c79996-jjcwx" event={"ID":"fa35d05a-b5c7-40b2-ac7a-6d72ebf85a69","Type":"ContainerStarted","Data":"d088a4b508be56432967f3915bfac92ded1fb7a06aa602853d0d7ce24719ce11"} Oct 30 18:38:16.808685 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:16.808636 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-65c8c79996-jjcwx" podStartSLOduration=2.496644873 podStartE2EDuration="4.808621951s" totalImagesPullingTime="2.311977078s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:38:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:16.806690969 +0000 UTC m=+264.335722192" watchObservedRunningTime="2026-10-30 18:38:16.808621951 +0000 UTC m=+264.337653089" Oct 30 18:38:17.773583 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:17.773552 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:18.161518 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:18.161486 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:20.596454 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:20.596422 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:20.968835 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:20.968761 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:23.765394 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:23.765351 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:23.931035 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:38:23.931005 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 18:38:24.339494 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:24.339445 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:25.824272 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:25.824229 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:26.256875 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:26.256783 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:26.712832 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:26.712802 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:26.830410 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:26.830381 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-ngggz"] Oct 30 18:38:26.839471 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:26.839445 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-ngggz" Oct 30 18:38:26.848809 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:26.848789 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljrdg\" (UniqueName: \"kubernetes.io/projected/3fff7b54-a723-42aa-ae6f-d5a3db247885-kube-api-access-ljrdg\") pod \"authorino-56c6c9ddd5-ngggz\" (UID: \"3fff7b54-a723-42aa-ae6f-d5a3db247885\") " pod="kuadrant-system/authorino-56c6c9ddd5-ngggz" Oct 30 18:38:26.851296 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:26.851273 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-ngggz"] Oct 30 18:38:26.855275 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:26.855257 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-255rx\"" Oct 30 18:38:26.880684 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:26.880660 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:26.949621 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:26.949597 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljrdg\" (UniqueName: \"kubernetes.io/projected/3fff7b54-a723-42aa-ae6f-d5a3db247885-kube-api-access-ljrdg\") pod \"authorino-56c6c9ddd5-ngggz\" (UID: \"3fff7b54-a723-42aa-ae6f-d5a3db247885\") " pod="kuadrant-system/authorino-56c6c9ddd5-ngggz" Oct 30 18:38:26.971838 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:26.971783 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljrdg\" (UniqueName: \"kubernetes.io/projected/3fff7b54-a723-42aa-ae6f-d5a3db247885-kube-api-access-ljrdg\") pod \"authorino-56c6c9ddd5-ngggz\" (UID: \"3fff7b54-a723-42aa-ae6f-d5a3db247885\") " pod="kuadrant-system/authorino-56c6c9ddd5-ngggz" Oct 30 18:38:27.150866 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:27.150848 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-ngggz" Oct 30 18:38:27.425080 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:27.425044 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-ngggz"] Oct 30 18:38:27.429857 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:38:27.429832 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3fff7b54_a723_42aa_ae6f_d5a3db247885.slice/crio-af07963f994ab806fd74542873e1f081b127f745f287dabf60afd6907e355555 WatchSource:0}: Error finding container af07963f994ab806fd74542873e1f081b127f745f287dabf60afd6907e355555: Status 404 returned error can't find the container with id af07963f994ab806fd74542873e1f081b127f745f287dabf60afd6907e355555 Oct 30 18:38:27.786193 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:27.786098 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-ngggz" event={"ID":"3fff7b54-a723-42aa-ae6f-d5a3db247885","Type":"ContainerStarted","Data":"af07963f994ab806fd74542873e1f081b127f745f287dabf60afd6907e355555"} Oct 30 18:38:29.300377 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:29.300352 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:29.418922 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:29.418875 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:29.758539 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:29.758510 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:29.797394 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:29.797365 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-ngggz" event={"ID":"3fff7b54-a723-42aa-ae6f-d5a3db247885","Type":"ContainerStarted","Data":"8d3d123c8da052a12fe5641574d0db2a28b046e5268c77ee005830f32372cf49"} Oct 30 18:38:29.823222 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:29.823129 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-ngggz" podStartSLOduration=2.005112157 podStartE2EDuration="3.823117325s" totalImagesPullingTime="1.818005168s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:38:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:29.822387504 +0000 UTC m=+277.351418642" watchObservedRunningTime="2026-10-30 18:38:29.823117325 +0000 UTC m=+277.352148464" Oct 30 18:38:29.954788 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:29.954746 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:30.157442 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:30.157412 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:30.466926 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:30.466844 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:35.689195 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:35.689097 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:35.812862 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:35.812828 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:36.603525 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:36.603489 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:38.305593 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:38.305558 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:38.452836 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:38.452805 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:49.709556 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:49.709519 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:53.930768 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:38:53.930735 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 18:38:55.243452 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:55.243415 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:55.815382 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:55.815348 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:55.880728 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:55.880694 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-549f447956-xm5vc"] Oct 30 18:38:55.913545 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:55.913505 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-549f447956-xm5vc"] Oct 30 18:38:55.913723 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:55.913610 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-549f447956-xm5vc" Oct 30 18:38:55.917098 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:55.917073 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 18:38:55.917262 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:55.917219 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 18:38:55.917759 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:55.917739 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 18:38:55.917759 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:55.917756 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 18:38:55.917895 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:55.917787 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 18:38:55.917895 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:55.917814 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-n582b\"" Oct 30 18:38:55.925680 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:55.925661 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:56.004484 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:56.004455 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2287ec66-a378-47f1-9daa-0e9ab9776050-webhook-cert\") pod \"maas-controller-549f447956-xm5vc\" (UID: \"2287ec66-a378-47f1-9daa-0e9ab9776050\") " pod="opendatahub/maas-controller-549f447956-xm5vc" Oct 30 18:38:56.004629 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:56.004490 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnj7k\" (UniqueName: \"kubernetes.io/projected/2287ec66-a378-47f1-9daa-0e9ab9776050-kube-api-access-vnj7k\") pod \"maas-controller-549f447956-xm5vc\" (UID: \"2287ec66-a378-47f1-9daa-0e9ab9776050\") " pod="opendatahub/maas-controller-549f447956-xm5vc" Oct 30 18:38:56.004629 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:56.004615 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2287ec66-a378-47f1-9daa-0e9ab9776050-metrics-certs\") pod \"maas-controller-549f447956-xm5vc\" (UID: \"2287ec66-a378-47f1-9daa-0e9ab9776050\") " pod="opendatahub/maas-controller-549f447956-xm5vc" Oct 30 18:38:56.033073 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:56.033038 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:56.106394 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:56.106311 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2287ec66-a378-47f1-9daa-0e9ab9776050-webhook-cert\") pod \"maas-controller-549f447956-xm5vc\" (UID: \"2287ec66-a378-47f1-9daa-0e9ab9776050\") " pod="opendatahub/maas-controller-549f447956-xm5vc" Oct 30 18:38:56.106394 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:56.106348 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnj7k\" (UniqueName: \"kubernetes.io/projected/2287ec66-a378-47f1-9daa-0e9ab9776050-kube-api-access-vnj7k\") pod \"maas-controller-549f447956-xm5vc\" (UID: \"2287ec66-a378-47f1-9daa-0e9ab9776050\") " pod="opendatahub/maas-controller-549f447956-xm5vc" Oct 30 18:38:56.106599 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:56.106434 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2287ec66-a378-47f1-9daa-0e9ab9776050-metrics-certs\") pod \"maas-controller-549f447956-xm5vc\" (UID: \"2287ec66-a378-47f1-9daa-0e9ab9776050\") " pod="opendatahub/maas-controller-549f447956-xm5vc" Oct 30 18:38:56.109088 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:56.109062 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2287ec66-a378-47f1-9daa-0e9ab9776050-webhook-cert\") pod \"maas-controller-549f447956-xm5vc\" (UID: \"2287ec66-a378-47f1-9daa-0e9ab9776050\") " pod="opendatahub/maas-controller-549f447956-xm5vc" Oct 30 18:38:56.109232 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:56.109129 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2287ec66-a378-47f1-9daa-0e9ab9776050-metrics-certs\") pod \"maas-controller-549f447956-xm5vc\" (UID: \"2287ec66-a378-47f1-9daa-0e9ab9776050\") " pod="opendatahub/maas-controller-549f447956-xm5vc" Oct 30 18:38:56.118989 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:56.118966 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnj7k\" (UniqueName: \"kubernetes.io/projected/2287ec66-a378-47f1-9daa-0e9ab9776050-kube-api-access-vnj7k\") pod \"maas-controller-549f447956-xm5vc\" (UID: \"2287ec66-a378-47f1-9daa-0e9ab9776050\") " pod="opendatahub/maas-controller-549f447956-xm5vc" Oct 30 18:38:56.138987 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:56.138965 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:56.226905 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:56.226880 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-549f447956-xm5vc" Oct 30 18:38:56.388035 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:56.387972 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-69dbdc8f84-kt9h9"] Oct 30 18:38:56.391956 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:56.391939 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-69dbdc8f84-kt9h9" Oct 30 18:38:56.403340 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:56.403321 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-69dbdc8f84-kt9h9"] Oct 30 18:38:56.511208 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:56.511182 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d18009e-009c-40be-9143-e468d1bdfc41-metrics-certs\") pod \"maas-controller-69dbdc8f84-kt9h9\" (UID: \"7d18009e-009c-40be-9143-e468d1bdfc41\") " pod="opendatahub/maas-controller-69dbdc8f84-kt9h9" Oct 30 18:38:56.511349 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:56.511251 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7d18009e-009c-40be-9143-e468d1bdfc41-webhook-cert\") pod \"maas-controller-69dbdc8f84-kt9h9\" (UID: \"7d18009e-009c-40be-9143-e468d1bdfc41\") " pod="opendatahub/maas-controller-69dbdc8f84-kt9h9" Oct 30 18:38:56.511349 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:56.511333 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q8r2\" (UniqueName: \"kubernetes.io/projected/7d18009e-009c-40be-9143-e468d1bdfc41-kube-api-access-5q8r2\") pod \"maas-controller-69dbdc8f84-kt9h9\" (UID: \"7d18009e-009c-40be-9143-e468d1bdfc41\") " pod="opendatahub/maas-controller-69dbdc8f84-kt9h9" Oct 30 18:38:56.511447 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:56.511384 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-549f447956-xm5vc"] Oct 30 18:38:56.514119 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:38:56.514093 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2287ec66_a378_47f1_9daa_0e9ab9776050.slice/crio-31a518cbcc325570823d18d4e96f70897c10ab29b37c88a76c29b073a6551371 WatchSource:0}: Error finding container 31a518cbcc325570823d18d4e96f70897c10ab29b37c88a76c29b073a6551371: Status 404 returned error can't find the container with id 31a518cbcc325570823d18d4e96f70897c10ab29b37c88a76c29b073a6551371 Oct 30 18:38:56.612722 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:56.612699 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q8r2\" (UniqueName: \"kubernetes.io/projected/7d18009e-009c-40be-9143-e468d1bdfc41-kube-api-access-5q8r2\") pod \"maas-controller-69dbdc8f84-kt9h9\" (UID: \"7d18009e-009c-40be-9143-e468d1bdfc41\") " pod="opendatahub/maas-controller-69dbdc8f84-kt9h9" Oct 30 18:38:56.612813 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:56.612782 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d18009e-009c-40be-9143-e468d1bdfc41-metrics-certs\") pod \"maas-controller-69dbdc8f84-kt9h9\" (UID: \"7d18009e-009c-40be-9143-e468d1bdfc41\") " pod="opendatahub/maas-controller-69dbdc8f84-kt9h9" Oct 30 18:38:56.612852 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:56.612836 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7d18009e-009c-40be-9143-e468d1bdfc41-webhook-cert\") pod \"maas-controller-69dbdc8f84-kt9h9\" (UID: \"7d18009e-009c-40be-9143-e468d1bdfc41\") " pod="opendatahub/maas-controller-69dbdc8f84-kt9h9" Oct 30 18:38:56.615301 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:56.615279 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7d18009e-009c-40be-9143-e468d1bdfc41-webhook-cert\") pod \"maas-controller-69dbdc8f84-kt9h9\" (UID: \"7d18009e-009c-40be-9143-e468d1bdfc41\") " pod="opendatahub/maas-controller-69dbdc8f84-kt9h9" Oct 30 18:38:56.615383 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:56.615336 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d18009e-009c-40be-9143-e468d1bdfc41-metrics-certs\") pod \"maas-controller-69dbdc8f84-kt9h9\" (UID: \"7d18009e-009c-40be-9143-e468d1bdfc41\") " pod="opendatahub/maas-controller-69dbdc8f84-kt9h9" Oct 30 18:38:56.624666 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:56.624649 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q8r2\" (UniqueName: \"kubernetes.io/projected/7d18009e-009c-40be-9143-e468d1bdfc41-kube-api-access-5q8r2\") pod \"maas-controller-69dbdc8f84-kt9h9\" (UID: \"7d18009e-009c-40be-9143-e468d1bdfc41\") " pod="opendatahub/maas-controller-69dbdc8f84-kt9h9" Oct 30 18:38:56.705719 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:56.705676 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-69dbdc8f84-kt9h9" Oct 30 18:38:56.917388 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:56.917358 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-549f447956-xm5vc" event={"ID":"2287ec66-a378-47f1-9daa-0e9ab9776050","Type":"ContainerStarted","Data":"31a518cbcc325570823d18d4e96f70897c10ab29b37c88a76c29b073a6551371"} Oct 30 18:38:56.980216 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:56.980194 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-69dbdc8f84-kt9h9"] Oct 30 18:38:56.982639 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:38:56.982613 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d18009e_009c_40be_9143_e468d1bdfc41.slice/crio-da12ce90c79799665fb11ce59e8304fd59f3cf689bd354b4f98ae81bb8ad1d53 WatchSource:0}: Error finding container da12ce90c79799665fb11ce59e8304fd59f3cf689bd354b4f98ae81bb8ad1d53: Status 404 returned error can't find the container with id da12ce90c79799665fb11ce59e8304fd59f3cf689bd354b4f98ae81bb8ad1d53 Oct 30 18:38:57.926527 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:38:57.926495 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-69dbdc8f84-kt9h9" event={"ID":"7d18009e-009c-40be-9143-e468d1bdfc41","Type":"ContainerStarted","Data":"da12ce90c79799665fb11ce59e8304fd59f3cf689bd354b4f98ae81bb8ad1d53"} Oct 30 18:39:00.715840 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:00.715800 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:00.839860 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:00.839824 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:00.943445 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:00.943404 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-549f447956-xm5vc" event={"ID":"2287ec66-a378-47f1-9daa-0e9ab9776050","Type":"ContainerStarted","Data":"bf6cc0cbe09da6152820942c10addb03a064c248dbd5dff9d0530c00c1036674"} Oct 30 18:39:00.943617 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:00.943580 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-549f447956-xm5vc" Oct 30 18:39:00.945259 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:00.945231 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-69dbdc8f84-kt9h9" event={"ID":"7d18009e-009c-40be-9143-e468d1bdfc41","Type":"ContainerStarted","Data":"f95331e99a3ab247b0b6667c00cd8b18e216a1934b04f72fb09169a25cb6b805"} Oct 30 18:39:00.945377 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:00.945364 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-69dbdc8f84-kt9h9" Oct 30 18:39:00.966789 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:00.966723 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-549f447956-xm5vc" podStartSLOduration=1.9896750189999999 podStartE2EDuration="5.966705808s" totalImagesPullingTime="3.977030789s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:38:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:39:00.964742281 +0000 UTC m=+308.493773422" watchObservedRunningTime="2026-10-30 18:39:00.966705808 +0000 UTC m=+308.495736943" Oct 30 18:39:01.014222 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:01.014169 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-69dbdc8f84-kt9h9" podStartSLOduration=1.495398753 podStartE2EDuration="5.014130557s" totalImagesPullingTime="3.518731804s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:38:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:39:01.013274477 +0000 UTC m=+308.542305614" watchObservedRunningTime="2026-10-30 18:39:01.014130557 +0000 UTC m=+308.543161697" Oct 30 18:39:01.698412 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:01.698377 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-k5ljv"] Oct 30 18:39:01.702682 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:01.702658 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-k5ljv" Oct 30 18:39:01.716664 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:01.715938 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Oct 30 18:39:01.716664 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:01.716014 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Oct 30 18:39:01.719063 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:01.719040 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-k5ljv"] Oct 30 18:39:01.719396 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:01.719331 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Oct 30 18:39:01.719540 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:01.719522 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-m4cvf\"" Oct 30 18:39:01.786051 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:01.786022 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:01.864354 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:01.864326 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5b43299a-0f21-4d69-8167-6f8739c40a9b-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-k5ljv\" (UID: \"5b43299a-0f21-4d69-8167-6f8739c40a9b\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-k5ljv" Oct 30 18:39:01.864502 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:01.864360 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/5b43299a-0f21-4d69-8167-6f8739c40a9b-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-k5ljv\" (UID: \"5b43299a-0f21-4d69-8167-6f8739c40a9b\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-k5ljv" Oct 30 18:39:01.864502 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:01.864390 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5b43299a-0f21-4d69-8167-6f8739c40a9b-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-k5ljv\" (UID: \"5b43299a-0f21-4d69-8167-6f8739c40a9b\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-k5ljv" Oct 30 18:39:01.864632 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:01.864510 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rc45\" (UniqueName: \"kubernetes.io/projected/5b43299a-0f21-4d69-8167-6f8739c40a9b-kube-api-access-7rc45\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-k5ljv\" (UID: \"5b43299a-0f21-4d69-8167-6f8739c40a9b\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-k5ljv" Oct 30 18:39:01.889649 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:01.889610 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:01.965099 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:01.965004 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5b43299a-0f21-4d69-8167-6f8739c40a9b-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-k5ljv\" (UID: \"5b43299a-0f21-4d69-8167-6f8739c40a9b\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-k5ljv" Oct 30 18:39:01.965099 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:01.965041 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/5b43299a-0f21-4d69-8167-6f8739c40a9b-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-k5ljv\" (UID: \"5b43299a-0f21-4d69-8167-6f8739c40a9b\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-k5ljv" Oct 30 18:39:01.965099 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:01.965083 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5b43299a-0f21-4d69-8167-6f8739c40a9b-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-k5ljv\" (UID: \"5b43299a-0f21-4d69-8167-6f8739c40a9b\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-k5ljv" Oct 30 18:39:01.965425 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:01.965135 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rc45\" (UniqueName: \"kubernetes.io/projected/5b43299a-0f21-4d69-8167-6f8739c40a9b-kube-api-access-7rc45\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-k5ljv\" (UID: \"5b43299a-0f21-4d69-8167-6f8739c40a9b\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-k5ljv" Oct 30 18:39:01.965887 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:01.965865 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5b43299a-0f21-4d69-8167-6f8739c40a9b-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-k5ljv\" (UID: \"5b43299a-0f21-4d69-8167-6f8739c40a9b\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-k5ljv" Oct 30 18:39:01.966101 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:01.966079 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/5b43299a-0f21-4d69-8167-6f8739c40a9b-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-k5ljv\" (UID: \"5b43299a-0f21-4d69-8167-6f8739c40a9b\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-k5ljv" Oct 30 18:39:01.967589 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:01.967566 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5b43299a-0f21-4d69-8167-6f8739c40a9b-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-k5ljv\" (UID: \"5b43299a-0f21-4d69-8167-6f8739c40a9b\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-k5ljv" Oct 30 18:39:01.978453 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:01.978434 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rc45\" (UniqueName: \"kubernetes.io/projected/5b43299a-0f21-4d69-8167-6f8739c40a9b-kube-api-access-7rc45\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-k5ljv\" (UID: \"5b43299a-0f21-4d69-8167-6f8739c40a9b\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-k5ljv" Oct 30 18:39:02.020452 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:02.020429 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-k5ljv" Oct 30 18:39:02.299002 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:02.298927 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-k5ljv"] Oct 30 18:39:02.301580 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:39:02.301548 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b43299a_0f21_4d69_8167_6f8739c40a9b.slice/crio-38bff09469ce5b49c7dbbd213f53b58c44dff4500cc267b506a7c3d4f82c022a WatchSource:0}: Error finding container 38bff09469ce5b49c7dbbd213f53b58c44dff4500cc267b506a7c3d4f82c022a: Status 404 returned error can't find the container with id 38bff09469ce5b49c7dbbd213f53b58c44dff4500cc267b506a7c3d4f82c022a Oct 30 18:39:02.955973 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:02.955943 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-k5ljv" event={"ID":"5b43299a-0f21-4d69-8167-6f8739c40a9b","Type":"ContainerStarted","Data":"38bff09469ce5b49c7dbbd213f53b58c44dff4500cc267b506a7c3d4f82c022a"} Oct 30 18:39:11.327630 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:11.327591 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:11.442811 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:11.442767 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-hwhlk"] Oct 30 18:39:11.467657 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:11.467619 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-hwhlk" Oct 30 18:39:11.477979 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:11.476723 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-hwhlk"] Oct 30 18:39:11.484809 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:11.483195 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-spxts"] Oct 30 18:39:11.487240 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:11.487221 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-spxts" Oct 30 18:39:11.488933 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:11.488901 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-bkw6n\"" Oct 30 18:39:11.489047 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:11.489033 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Oct 30 18:39:11.490592 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:11.490501 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:11.503464 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:11.503442 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-spxts"] Oct 30 18:39:11.546306 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:11.546279 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-mkv7p\"" Oct 30 18:39:11.546450 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:11.546364 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Oct 30 18:39:11.561505 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:11.561479 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x44vx\" (UniqueName: \"kubernetes.io/projected/d99c9271-c3ea-4f9b-8668-c0348c766af0-kube-api-access-x44vx\") pod \"model-serving-api-75bd89fdfc-spxts\" (UID: \"d99c9271-c3ea-4f9b-8668-c0348c766af0\") " pod="opendatahub/model-serving-api-75bd89fdfc-spxts" Oct 30 18:39:11.561642 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:11.561566 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgpv6\" (UniqueName: \"kubernetes.io/projected/37cb80da-410f-4ff5-98e2-34eae144104f-kube-api-access-qgpv6\") pod \"llmisvc-controller-manager-869d5dcc5d-hwhlk\" (UID: \"37cb80da-410f-4ff5-98e2-34eae144104f\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-hwhlk" Oct 30 18:39:11.561698 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:11.561669 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/37cb80da-410f-4ff5-98e2-34eae144104f-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-hwhlk\" (UID: \"37cb80da-410f-4ff5-98e2-34eae144104f\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-hwhlk" Oct 30 18:39:11.561747 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:11.561714 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d99c9271-c3ea-4f9b-8668-c0348c766af0-tls-certs\") pod \"model-serving-api-75bd89fdfc-spxts\" (UID: \"d99c9271-c3ea-4f9b-8668-c0348c766af0\") " pod="opendatahub/model-serving-api-75bd89fdfc-spxts" Oct 30 18:39:11.641664 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:11.641578 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:11.662883 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:11.662850 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x44vx\" (UniqueName: \"kubernetes.io/projected/d99c9271-c3ea-4f9b-8668-c0348c766af0-kube-api-access-x44vx\") pod \"model-serving-api-75bd89fdfc-spxts\" (UID: \"d99c9271-c3ea-4f9b-8668-c0348c766af0\") " pod="opendatahub/model-serving-api-75bd89fdfc-spxts" Oct 30 18:39:11.663022 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:11.662920 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgpv6\" (UniqueName: \"kubernetes.io/projected/37cb80da-410f-4ff5-98e2-34eae144104f-kube-api-access-qgpv6\") pod \"llmisvc-controller-manager-869d5dcc5d-hwhlk\" (UID: \"37cb80da-410f-4ff5-98e2-34eae144104f\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-hwhlk" Oct 30 18:39:11.663080 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:11.663026 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/37cb80da-410f-4ff5-98e2-34eae144104f-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-hwhlk\" (UID: \"37cb80da-410f-4ff5-98e2-34eae144104f\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-hwhlk" Oct 30 18:39:11.663122 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:11.663089 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d99c9271-c3ea-4f9b-8668-c0348c766af0-tls-certs\") pod \"model-serving-api-75bd89fdfc-spxts\" (UID: \"d99c9271-c3ea-4f9b-8668-c0348c766af0\") " pod="opendatahub/model-serving-api-75bd89fdfc-spxts" Oct 30 18:39:11.666002 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:11.665969 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/37cb80da-410f-4ff5-98e2-34eae144104f-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-hwhlk\" (UID: \"37cb80da-410f-4ff5-98e2-34eae144104f\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-hwhlk" Oct 30 18:39:11.666168 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:11.666118 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d99c9271-c3ea-4f9b-8668-c0348c766af0-tls-certs\") pod \"model-serving-api-75bd89fdfc-spxts\" (UID: \"d99c9271-c3ea-4f9b-8668-c0348c766af0\") " pod="opendatahub/model-serving-api-75bd89fdfc-spxts" Oct 30 18:39:11.690180 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:11.677364 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgpv6\" (UniqueName: \"kubernetes.io/projected/37cb80da-410f-4ff5-98e2-34eae144104f-kube-api-access-qgpv6\") pod \"llmisvc-controller-manager-869d5dcc5d-hwhlk\" (UID: \"37cb80da-410f-4ff5-98e2-34eae144104f\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-hwhlk" Oct 30 18:39:11.690180 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:11.679494 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x44vx\" (UniqueName: \"kubernetes.io/projected/d99c9271-c3ea-4f9b-8668-c0348c766af0-kube-api-access-x44vx\") pod \"model-serving-api-75bd89fdfc-spxts\" (UID: \"d99c9271-c3ea-4f9b-8668-c0348c766af0\") " pod="opendatahub/model-serving-api-75bd89fdfc-spxts" Oct 30 18:39:11.751262 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:11.751227 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:11.787252 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:11.787214 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-hwhlk" Oct 30 18:39:11.807067 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:11.807040 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-spxts" Oct 30 18:39:11.953040 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:11.952967 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-549f447956-xm5vc" Oct 30 18:39:11.953302 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:11.953215 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-69dbdc8f84-kt9h9" Oct 30 18:39:12.108383 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:12.108226 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-hwhlk"] Oct 30 18:39:12.111457 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:39:12.111428 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod37cb80da_410f_4ff5_98e2_34eae144104f.slice/crio-ed339f319a466c92e4a6fea626c5d29ad733aed78623ba7557bfdf684a16cd7a WatchSource:0}: Error finding container ed339f319a466c92e4a6fea626c5d29ad733aed78623ba7557bfdf684a16cd7a: Status 404 returned error can't find the container with id ed339f319a466c92e4a6fea626c5d29ad733aed78623ba7557bfdf684a16cd7a Oct 30 18:39:12.114566 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:12.114532 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-549f447956-xm5vc"] Oct 30 18:39:12.114737 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:12.114714 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-549f447956-xm5vc" podUID="2287ec66-a378-47f1-9daa-0e9ab9776050" containerName="manager" containerID="cri-o://bf6cc0cbe09da6152820942c10addb03a064c248dbd5dff9d0530c00c1036674" gracePeriod=30 Oct 30 18:39:12.131810 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:12.131793 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-spxts"] Oct 30 18:39:12.182300 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:39:12.182255 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd99c9271_c3ea_4f9b_8668_c0348c766af0.slice/crio-433cc74e9b90de9d4f3d12bda6674836b36fcf1b2eab7b035936f527f61bfc59 WatchSource:0}: Error finding container 433cc74e9b90de9d4f3d12bda6674836b36fcf1b2eab7b035936f527f61bfc59: Status 404 returned error can't find the container with id 433cc74e9b90de9d4f3d12bda6674836b36fcf1b2eab7b035936f527f61bfc59 Oct 30 18:39:12.358225 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:12.358198 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-549f447956-xm5vc" Oct 30 18:39:12.388587 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:12.388543 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:12.472668 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:12.472581 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2287ec66-a378-47f1-9daa-0e9ab9776050-webhook-cert\" (UniqueName: \"kubernetes.io/secret/2287ec66-a378-47f1-9daa-0e9ab9776050-webhook-cert\") pod \"2287ec66-a378-47f1-9daa-0e9ab9776050\" (UID: \"2287ec66-a378-47f1-9daa-0e9ab9776050\") " Oct 30 18:39:12.472847 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:12.472692 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2287ec66-a378-47f1-9daa-0e9ab9776050-kube-api-access-vnj7k\" (UniqueName: \"kubernetes.io/projected/2287ec66-a378-47f1-9daa-0e9ab9776050-kube-api-access-vnj7k\") pod \"2287ec66-a378-47f1-9daa-0e9ab9776050\" (UID: \"2287ec66-a378-47f1-9daa-0e9ab9776050\") " Oct 30 18:39:12.472847 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:12.472727 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2287ec66-a378-47f1-9daa-0e9ab9776050-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2287ec66-a378-47f1-9daa-0e9ab9776050-metrics-certs\") pod \"2287ec66-a378-47f1-9daa-0e9ab9776050\" (UID: \"2287ec66-a378-47f1-9daa-0e9ab9776050\") " Oct 30 18:39:12.475359 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:12.475313 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2287ec66-a378-47f1-9daa-0e9ab9776050-metrics-certs" pod "2287ec66-a378-47f1-9daa-0e9ab9776050" (UID: "2287ec66-a378-47f1-9daa-0e9ab9776050"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:39:12.475489 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:12.475426 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2287ec66-a378-47f1-9daa-0e9ab9776050-webhook-cert" pod "2287ec66-a378-47f1-9daa-0e9ab9776050" (UID: "2287ec66-a378-47f1-9daa-0e9ab9776050"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:39:12.475489 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:12.475464 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2287ec66-a378-47f1-9daa-0e9ab9776050-kube-api-access-vnj7k" pod "2287ec66-a378-47f1-9daa-0e9ab9776050" (UID: "2287ec66-a378-47f1-9daa-0e9ab9776050"). InnerVolumeSpecName "kube-api-access-vnj7k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:39:12.508042 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:12.508009 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:12.573824 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:12.573791 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnj7k\" (UniqueName: \"kubernetes.io/projected/2287ec66-a378-47f1-9daa-0e9ab9776050-kube-api-access-vnj7k\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:39:12.573824 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:12.573823 6960 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2287ec66-a378-47f1-9daa-0e9ab9776050-metrics-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:39:12.574066 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:12.573834 6960 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2287ec66-a378-47f1-9daa-0e9ab9776050-webhook-cert\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:39:12.788827 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:12.788699 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:12.972857 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:12.972492 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:13.020436 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:13.020397 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-spxts" event={"ID":"d99c9271-c3ea-4f9b-8668-c0348c766af0","Type":"ContainerStarted","Data":"433cc74e9b90de9d4f3d12bda6674836b36fcf1b2eab7b035936f527f61bfc59"} Oct 30 18:39:13.022707 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:13.022649 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-hwhlk" event={"ID":"37cb80da-410f-4ff5-98e2-34eae144104f","Type":"ContainerStarted","Data":"ed339f319a466c92e4a6fea626c5d29ad733aed78623ba7557bfdf684a16cd7a"} Oct 30 18:39:13.025180 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:13.025129 6960 generic.go:356] "Generic (PLEG): container finished" podID="2287ec66-a378-47f1-9daa-0e9ab9776050" containerID="bf6cc0cbe09da6152820942c10addb03a064c248dbd5dff9d0530c00c1036674" exitCode=0 Oct 30 18:39:13.025312 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:13.025191 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-549f447956-xm5vc" event={"ID":"2287ec66-a378-47f1-9daa-0e9ab9776050","Type":"ContainerDied","Data":"bf6cc0cbe09da6152820942c10addb03a064c248dbd5dff9d0530c00c1036674"} Oct 30 18:39:13.025312 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:13.025229 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-549f447956-xm5vc" event={"ID":"2287ec66-a378-47f1-9daa-0e9ab9776050","Type":"ContainerDied","Data":"31a518cbcc325570823d18d4e96f70897c10ab29b37c88a76c29b073a6551371"} Oct 30 18:39:13.025312 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:13.025258 6960 scope.go:122] "RemoveContainer" containerID="bf6cc0cbe09da6152820942c10addb03a064c248dbd5dff9d0530c00c1036674" Oct 30 18:39:13.025479 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:13.025404 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-549f447956-xm5vc" Oct 30 18:39:13.038908 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:13.038857 6960 scope.go:122] "RemoveContainer" containerID="bf6cc0cbe09da6152820942c10addb03a064c248dbd5dff9d0530c00c1036674" Oct 30 18:39:13.039712 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:39:13.039690 6960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf6cc0cbe09da6152820942c10addb03a064c248dbd5dff9d0530c00c1036674\": container with ID starting with bf6cc0cbe09da6152820942c10addb03a064c248dbd5dff9d0530c00c1036674 not found: ID does not exist" containerID="bf6cc0cbe09da6152820942c10addb03a064c248dbd5dff9d0530c00c1036674" Oct 30 18:39:13.039801 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:13.039721 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf6cc0cbe09da6152820942c10addb03a064c248dbd5dff9d0530c00c1036674"} err="failed to get container status \"bf6cc0cbe09da6152820942c10addb03a064c248dbd5dff9d0530c00c1036674\": rpc error: code = NotFound desc = could not find container \"bf6cc0cbe09da6152820942c10addb03a064c248dbd5dff9d0530c00c1036674\": container with ID starting with bf6cc0cbe09da6152820942c10addb03a064c248dbd5dff9d0530c00c1036674 not found: ID does not exist" Oct 30 18:39:13.047613 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:13.047573 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-549f447956-xm5vc"] Oct 30 18:39:13.053808 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:13.053771 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-549f447956-xm5vc"] Oct 30 18:39:13.086759 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:13.086633 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:14.594231 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:14.594193 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2287ec66-a378-47f1-9daa-0e9ab9776050" path="/var/lib/kubelet/pods/2287ec66-a378-47f1-9daa-0e9ab9776050/volumes" Oct 30 18:39:15.041701 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:15.041617 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-hwhlk" event={"ID":"37cb80da-410f-4ff5-98e2-34eae144104f","Type":"ContainerStarted","Data":"e91549592797bdfd41dbe95a747c11d0d411a0688e1857468e58720da7d41e5d"} Oct 30 18:39:15.041880 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:15.041742 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-hwhlk" Oct 30 18:39:15.043633 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:15.043607 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-spxts" event={"ID":"d99c9271-c3ea-4f9b-8668-c0348c766af0","Type":"ContainerStarted","Data":"9edad317098a6f1a79aaebc7c4bf190f7cf88eda44f1849673a12a5922d12437"} Oct 30 18:39:15.043749 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:15.043699 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-spxts" Oct 30 18:39:15.061580 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:15.061516 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-hwhlk" podStartSLOduration=1.441572128 podStartE2EDuration="4.061498919s" totalImagesPullingTime="2.619926791s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:39:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:39:15.059630435 +0000 UTC m=+322.588661574" watchObservedRunningTime="2026-10-30 18:39:15.061498919 +0000 UTC m=+322.590530056" Oct 30 18:39:15.079403 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:15.079340 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-spxts" podStartSLOduration=1.919288332 podStartE2EDuration="4.079318812s" totalImagesPullingTime="2.16003048s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:39:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:39:15.077192285 +0000 UTC m=+322.606223426" watchObservedRunningTime="2026-10-30 18:39:15.079318812 +0000 UTC m=+322.608349953" Oct 30 18:39:16.049607 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:16.049568 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-k5ljv" event={"ID":"5b43299a-0f21-4d69-8167-6f8739c40a9b","Type":"ContainerStarted","Data":"e15ed55bdc4d79aeceb897199f5add8cb8b6e093a4726aad8aa8b7e621fc7106"} Oct 30 18:39:16.050017 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:16.049928 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-k5ljv" Oct 30 18:39:16.071075 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:16.071029 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-k5ljv" podStartSLOduration=1.9454816579999998 podStartE2EDuration="15.071016892s" totalImagesPullingTime="13.125535234s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:39:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:39:16.068785758 +0000 UTC m=+323.597816896" watchObservedRunningTime="2026-10-30 18:39:16.071016892 +0000 UTC m=+323.600048030" Oct 30 18:39:23.931181 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:39:23.931121 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 18:39:26.053733 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:26.053702 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-spxts" Oct 30 18:39:27.059713 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:27.059687 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-k5ljv" Oct 30 18:39:32.168693 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:32.168653 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:32.385218 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:32.385181 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:33.443188 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:33.443137 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:33.555244 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:33.555202 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:33.657791 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:33.657763 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6747cfd45c-zglnn"] Oct 30 18:39:33.658198 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:33.658184 6960 cpu_manager.go:414] "containerMap: removing container" podUID="2287ec66-a378-47f1-9daa-0e9ab9776050" containerName="manager" Oct 30 18:39:33.658247 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:33.658199 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2287ec66-a378-47f1-9daa-0e9ab9776050" containerName="manager" Oct 30 18:39:33.658306 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:33.658298 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="2287ec66-a378-47f1-9daa-0e9ab9776050" containerName="manager" Oct 30 18:39:33.664006 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:33.663981 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6747cfd45c-zglnn" Oct 30 18:39:33.674681 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:33.674654 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6747cfd45c-zglnn"] Oct 30 18:39:33.675380 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:33.675355 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:39:33.675380 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:33.675377 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:39:33.675613 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:33.675453 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 18:39:33.675613 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:33.675486 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-plgsl\"" Oct 30 18:39:33.675613 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:33.675501 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 30 18:39:33.676729 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:33.676705 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:33.767818 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:33.767748 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9fa2248-8e6f-413b-858b-ca0b9a363765-metrics-certs\") pod \"maas-api-6747cfd45c-zglnn\" (UID: \"c9fa2248-8e6f-413b-858b-ca0b9a363765\") " pod="odh-ai-gateway-infra/maas-api-6747cfd45c-zglnn" Oct 30 18:39:33.767818 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:33.767781 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9fa2248-8e6f-413b-858b-ca0b9a363765-maas-api-tls\") pod \"maas-api-6747cfd45c-zglnn\" (UID: \"c9fa2248-8e6f-413b-858b-ca0b9a363765\") " pod="odh-ai-gateway-infra/maas-api-6747cfd45c-zglnn" Oct 30 18:39:33.767818 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:33.767807 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfkwm\" (UniqueName: \"kubernetes.io/projected/c9fa2248-8e6f-413b-858b-ca0b9a363765-kube-api-access-sfkwm\") pod \"maas-api-6747cfd45c-zglnn\" (UID: \"c9fa2248-8e6f-413b-858b-ca0b9a363765\") " pod="odh-ai-gateway-infra/maas-api-6747cfd45c-zglnn" Oct 30 18:39:33.768020 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:33.767941 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9fa2248-8e6f-413b-858b-ca0b9a363765-openshift-service-ca\") pod \"maas-api-6747cfd45c-zglnn\" (UID: \"c9fa2248-8e6f-413b-858b-ca0b9a363765\") " pod="odh-ai-gateway-infra/maas-api-6747cfd45c-zglnn" Oct 30 18:39:33.871815 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:33.869647 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9fa2248-8e6f-413b-858b-ca0b9a363765-metrics-certs\") pod \"maas-api-6747cfd45c-zglnn\" (UID: \"c9fa2248-8e6f-413b-858b-ca0b9a363765\") " pod="odh-ai-gateway-infra/maas-api-6747cfd45c-zglnn" Oct 30 18:39:33.871815 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:33.869723 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9fa2248-8e6f-413b-858b-ca0b9a363765-maas-api-tls\") pod \"maas-api-6747cfd45c-zglnn\" (UID: \"c9fa2248-8e6f-413b-858b-ca0b9a363765\") " pod="odh-ai-gateway-infra/maas-api-6747cfd45c-zglnn" Oct 30 18:39:33.871815 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:33.869770 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfkwm\" (UniqueName: \"kubernetes.io/projected/c9fa2248-8e6f-413b-858b-ca0b9a363765-kube-api-access-sfkwm\") pod \"maas-api-6747cfd45c-zglnn\" (UID: \"c9fa2248-8e6f-413b-858b-ca0b9a363765\") " pod="odh-ai-gateway-infra/maas-api-6747cfd45c-zglnn" Oct 30 18:39:33.871815 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:33.869841 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9fa2248-8e6f-413b-858b-ca0b9a363765-openshift-service-ca\") pod \"maas-api-6747cfd45c-zglnn\" (UID: \"c9fa2248-8e6f-413b-858b-ca0b9a363765\") " pod="odh-ai-gateway-infra/maas-api-6747cfd45c-zglnn" Oct 30 18:39:33.871815 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:33.870730 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9fa2248-8e6f-413b-858b-ca0b9a363765-openshift-service-ca\") pod \"maas-api-6747cfd45c-zglnn\" (UID: \"c9fa2248-8e6f-413b-858b-ca0b9a363765\") " pod="odh-ai-gateway-infra/maas-api-6747cfd45c-zglnn" Oct 30 18:39:33.874126 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:33.874095 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9fa2248-8e6f-413b-858b-ca0b9a363765-metrics-certs\") pod \"maas-api-6747cfd45c-zglnn\" (UID: \"c9fa2248-8e6f-413b-858b-ca0b9a363765\") " pod="odh-ai-gateway-infra/maas-api-6747cfd45c-zglnn" Oct 30 18:39:33.876838 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:33.876813 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9fa2248-8e6f-413b-858b-ca0b9a363765-maas-api-tls\") pod \"maas-api-6747cfd45c-zglnn\" (UID: \"c9fa2248-8e6f-413b-858b-ca0b9a363765\") " pod="odh-ai-gateway-infra/maas-api-6747cfd45c-zglnn" Oct 30 18:39:33.892624 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:33.892600 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfkwm\" (UniqueName: \"kubernetes.io/projected/c9fa2248-8e6f-413b-858b-ca0b9a363765-kube-api-access-sfkwm\") pod \"maas-api-6747cfd45c-zglnn\" (UID: \"c9fa2248-8e6f-413b-858b-ca0b9a363765\") " pod="odh-ai-gateway-infra/maas-api-6747cfd45c-zglnn" Oct 30 18:39:33.978935 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:33.978903 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6747cfd45c-zglnn" Oct 30 18:39:34.266167 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:34.266096 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6747cfd45c-zglnn"] Oct 30 18:39:34.272541 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:39:34.272509 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9fa2248_8e6f_413b_858b_ca0b9a363765.slice/crio-b8fc15adabea8100887cad0c3be813c00c97bccca52c2266859a4c7967941993 WatchSource:0}: Error finding container b8fc15adabea8100887cad0c3be813c00c97bccca52c2266859a4c7967941993: Status 404 returned error can't find the container with id b8fc15adabea8100887cad0c3be813c00c97bccca52c2266859a4c7967941993 Oct 30 18:39:34.274006 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:34.273990 6960 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 18:39:35.144302 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:35.144270 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6747cfd45c-zglnn" event={"ID":"c9fa2248-8e6f-413b-858b-ca0b9a363765","Type":"ContainerStarted","Data":"b8fc15adabea8100887cad0c3be813c00c97bccca52c2266859a4c7967941993"} Oct 30 18:39:38.159744 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:38.159714 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6747cfd45c-zglnn" event={"ID":"c9fa2248-8e6f-413b-858b-ca0b9a363765","Type":"ContainerStarted","Data":"2cf139c4968e996473b6e2e0ab62c721a05c070417031b8a124f3d6b40b4bd24"} Oct 30 18:39:38.160103 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:38.159817 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6747cfd45c-zglnn" Oct 30 18:39:38.181874 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:38.181787 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6747cfd45c-zglnn" podStartSLOduration=1.530275651 podStartE2EDuration="5.181773517s" totalImagesPullingTime="3.651497866s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:39:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:39:38.179796235 +0000 UTC m=+345.708827373" watchObservedRunningTime="2026-10-30 18:39:38.181773517 +0000 UTC m=+345.710804655" Oct 30 18:39:44.168870 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:44.168846 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6747cfd45c-zglnn" Oct 30 18:39:44.653166 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:44.653105 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:44.988276 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:44.988175 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:45.105082 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:45.105037 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:46.052133 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:46.052106 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-hwhlk" Oct 30 18:39:48.178196 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:48.178165 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-ngggz"] Oct 30 18:39:48.178622 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:48.178342 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-ngggz" podUID="3fff7b54-a723-42aa-ae6f-d5a3db247885" containerName="authorino" containerID="cri-o://8d3d123c8da052a12fe5641574d0db2a28b046e5268c77ee005830f32372cf49" gracePeriod=30 Oct 30 18:39:48.426845 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:48.426817 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-ngggz" Oct 30 18:39:48.511733 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:48.511652 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fff7b54-a723-42aa-ae6f-d5a3db247885-kube-api-access-ljrdg\" (UniqueName: \"kubernetes.io/projected/3fff7b54-a723-42aa-ae6f-d5a3db247885-kube-api-access-ljrdg\") pod \"3fff7b54-a723-42aa-ae6f-d5a3db247885\" (UID: \"3fff7b54-a723-42aa-ae6f-d5a3db247885\") " Oct 30 18:39:48.513979 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:48.513949 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fff7b54-a723-42aa-ae6f-d5a3db247885-kube-api-access-ljrdg" pod "3fff7b54-a723-42aa-ae6f-d5a3db247885" (UID: "3fff7b54-a723-42aa-ae6f-d5a3db247885"). InnerVolumeSpecName "kube-api-access-ljrdg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:39:48.612950 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:48.612925 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljrdg\" (UniqueName: \"kubernetes.io/projected/3fff7b54-a723-42aa-ae6f-d5a3db247885-kube-api-access-ljrdg\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:39:49.215218 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:49.215188 6960 generic.go:356] "Generic (PLEG): container finished" podID="3fff7b54-a723-42aa-ae6f-d5a3db247885" containerID="8d3d123c8da052a12fe5641574d0db2a28b046e5268c77ee005830f32372cf49" exitCode=0 Oct 30 18:39:49.215646 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:49.215246 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-ngggz" Oct 30 18:39:49.215646 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:49.215281 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-ngggz" event={"ID":"3fff7b54-a723-42aa-ae6f-d5a3db247885","Type":"ContainerDied","Data":"8d3d123c8da052a12fe5641574d0db2a28b046e5268c77ee005830f32372cf49"} Oct 30 18:39:49.215646 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:49.215323 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-ngggz" event={"ID":"3fff7b54-a723-42aa-ae6f-d5a3db247885","Type":"ContainerDied","Data":"af07963f994ab806fd74542873e1f081b127f745f287dabf60afd6907e355555"} Oct 30 18:39:49.215646 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:49.215347 6960 scope.go:122] "RemoveContainer" containerID="8d3d123c8da052a12fe5641574d0db2a28b046e5268c77ee005830f32372cf49" Oct 30 18:39:49.224158 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:49.224129 6960 scope.go:122] "RemoveContainer" containerID="8d3d123c8da052a12fe5641574d0db2a28b046e5268c77ee005830f32372cf49" Oct 30 18:39:49.224436 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:39:49.224417 6960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d3d123c8da052a12fe5641574d0db2a28b046e5268c77ee005830f32372cf49\": container with ID starting with 8d3d123c8da052a12fe5641574d0db2a28b046e5268c77ee005830f32372cf49 not found: ID does not exist" containerID="8d3d123c8da052a12fe5641574d0db2a28b046e5268c77ee005830f32372cf49" Oct 30 18:39:49.224484 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:49.224449 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d3d123c8da052a12fe5641574d0db2a28b046e5268c77ee005830f32372cf49"} err="failed to get container status \"8d3d123c8da052a12fe5641574d0db2a28b046e5268c77ee005830f32372cf49\": rpc error: code = NotFound desc = could not find container \"8d3d123c8da052a12fe5641574d0db2a28b046e5268c77ee005830f32372cf49\": container with ID starting with 8d3d123c8da052a12fe5641574d0db2a28b046e5268c77ee005830f32372cf49 not found: ID does not exist" Oct 30 18:39:49.231390 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:49.231368 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-ngggz"] Oct 30 18:39:49.235761 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:49.235742 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-ngggz"] Oct 30 18:39:49.759450 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:49.759417 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:50.591640 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:50.591611 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fff7b54-a723-42aa-ae6f-d5a3db247885" path="/var/lib/kubelet/pods/3fff7b54-a723-42aa-ae6f-d5a3db247885/volumes" Oct 30 18:39:53.931067 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:39:53.931026 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 18:39:55.213299 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:55.213267 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6747cfd45c-zglnn"] Oct 30 18:39:55.213726 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:55.213464 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6747cfd45c-zglnn" podUID="c9fa2248-8e6f-413b-858b-ca0b9a363765" containerName="maas-api" containerID="cri-o://2cf139c4968e996473b6e2e0ab62c721a05c070417031b8a124f3d6b40b4bd24" gracePeriod=30 Oct 30 18:39:55.456734 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:55.456708 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6747cfd45c-zglnn" Oct 30 18:39:55.469719 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:55.469648 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9fa2248-8e6f-413b-858b-ca0b9a363765-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9fa2248-8e6f-413b-858b-ca0b9a363765-openshift-service-ca\") pod \"c9fa2248-8e6f-413b-858b-ca0b9a363765\" (UID: \"c9fa2248-8e6f-413b-858b-ca0b9a363765\") " Oct 30 18:39:55.469719 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:55.469699 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9fa2248-8e6f-413b-858b-ca0b9a363765-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9fa2248-8e6f-413b-858b-ca0b9a363765-metrics-certs\") pod \"c9fa2248-8e6f-413b-858b-ca0b9a363765\" (UID: \"c9fa2248-8e6f-413b-858b-ca0b9a363765\") " Oct 30 18:39:55.469929 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:55.469813 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9fa2248-8e6f-413b-858b-ca0b9a363765-kube-api-access-sfkwm\" (UniqueName: \"kubernetes.io/projected/c9fa2248-8e6f-413b-858b-ca0b9a363765-kube-api-access-sfkwm\") pod \"c9fa2248-8e6f-413b-858b-ca0b9a363765\" (UID: \"c9fa2248-8e6f-413b-858b-ca0b9a363765\") " Oct 30 18:39:55.469929 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:55.469882 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9fa2248-8e6f-413b-858b-ca0b9a363765-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9fa2248-8e6f-413b-858b-ca0b9a363765-maas-api-tls\") pod \"c9fa2248-8e6f-413b-858b-ca0b9a363765\" (UID: \"c9fa2248-8e6f-413b-858b-ca0b9a363765\") " Oct 30 18:39:55.470039 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:55.469986 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9fa2248-8e6f-413b-858b-ca0b9a363765-openshift-service-ca" pod "c9fa2248-8e6f-413b-858b-ca0b9a363765" (UID: "c9fa2248-8e6f-413b-858b-ca0b9a363765"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:39:55.470223 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:55.470197 6960 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9fa2248-8e6f-413b-858b-ca0b9a363765-openshift-service-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:39:55.472240 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:55.472127 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9fa2248-8e6f-413b-858b-ca0b9a363765-maas-api-tls" pod "c9fa2248-8e6f-413b-858b-ca0b9a363765" (UID: "c9fa2248-8e6f-413b-858b-ca0b9a363765"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:39:55.472240 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:55.472220 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9fa2248-8e6f-413b-858b-ca0b9a363765-kube-api-access-sfkwm" pod "c9fa2248-8e6f-413b-858b-ca0b9a363765" (UID: "c9fa2248-8e6f-413b-858b-ca0b9a363765"). InnerVolumeSpecName "kube-api-access-sfkwm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:39:55.472404 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:55.472192 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9fa2248-8e6f-413b-858b-ca0b9a363765-metrics-certs" pod "c9fa2248-8e6f-413b-858b-ca0b9a363765" (UID: "c9fa2248-8e6f-413b-858b-ca0b9a363765"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:39:55.570693 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:55.570662 6960 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9fa2248-8e6f-413b-858b-ca0b9a363765-metrics-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:39:55.570693 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:55.570690 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfkwm\" (UniqueName: \"kubernetes.io/projected/c9fa2248-8e6f-413b-858b-ca0b9a363765-kube-api-access-sfkwm\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:39:55.570693 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:55.570701 6960 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9fa2248-8e6f-413b-858b-ca0b9a363765-maas-api-tls\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:39:56.251019 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:56.250982 6960 generic.go:356] "Generic (PLEG): container finished" podID="c9fa2248-8e6f-413b-858b-ca0b9a363765" containerID="2cf139c4968e996473b6e2e0ab62c721a05c070417031b8a124f3d6b40b4bd24" exitCode=0 Oct 30 18:39:56.251479 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:56.251027 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6747cfd45c-zglnn" Oct 30 18:39:56.251479 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:56.251033 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6747cfd45c-zglnn" event={"ID":"c9fa2248-8e6f-413b-858b-ca0b9a363765","Type":"ContainerDied","Data":"2cf139c4968e996473b6e2e0ab62c721a05c070417031b8a124f3d6b40b4bd24"} Oct 30 18:39:56.251479 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:56.251088 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6747cfd45c-zglnn" event={"ID":"c9fa2248-8e6f-413b-858b-ca0b9a363765","Type":"ContainerDied","Data":"b8fc15adabea8100887cad0c3be813c00c97bccca52c2266859a4c7967941993"} Oct 30 18:39:56.251479 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:56.251117 6960 scope.go:122] "RemoveContainer" containerID="2cf139c4968e996473b6e2e0ab62c721a05c070417031b8a124f3d6b40b4bd24" Oct 30 18:39:56.261534 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:56.261509 6960 scope.go:122] "RemoveContainer" containerID="2cf139c4968e996473b6e2e0ab62c721a05c070417031b8a124f3d6b40b4bd24" Oct 30 18:39:56.261889 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:39:56.261869 6960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cf139c4968e996473b6e2e0ab62c721a05c070417031b8a124f3d6b40b4bd24\": container with ID starting with 2cf139c4968e996473b6e2e0ab62c721a05c070417031b8a124f3d6b40b4bd24 not found: ID does not exist" containerID="2cf139c4968e996473b6e2e0ab62c721a05c070417031b8a124f3d6b40b4bd24" Oct 30 18:39:56.261983 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:56.261900 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cf139c4968e996473b6e2e0ab62c721a05c070417031b8a124f3d6b40b4bd24"} err="failed to get container status \"2cf139c4968e996473b6e2e0ab62c721a05c070417031b8a124f3d6b40b4bd24\": rpc error: code = NotFound desc = could not find container \"2cf139c4968e996473b6e2e0ab62c721a05c070417031b8a124f3d6b40b4bd24\": container with ID starting with 2cf139c4968e996473b6e2e0ab62c721a05c070417031b8a124f3d6b40b4bd24 not found: ID does not exist" Oct 30 18:39:56.275622 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:56.275593 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6747cfd45c-zglnn"] Oct 30 18:39:56.281752 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:56.281728 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6747cfd45c-zglnn"] Oct 30 18:39:56.591840 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:56.591810 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9fa2248-8e6f-413b-858b-ca0b9a363765" path="/var/lib/kubelet/pods/c9fa2248-8e6f-413b-858b-ca0b9a363765/volumes" Oct 30 18:39:59.891194 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:39:59.891136 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:00.002337 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:40:00.002303 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:09.429011 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:40:09.428930 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-xkxkl"] Oct 30 18:40:09.429507 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:40:09.429414 6960 cpu_manager.go:414] "containerMap: removing container" podUID="3fff7b54-a723-42aa-ae6f-d5a3db247885" containerName="authorino" Oct 30 18:40:09.429507 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:40:09.429428 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fff7b54-a723-42aa-ae6f-d5a3db247885" containerName="authorino" Oct 30 18:40:09.429507 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:40:09.429465 6960 cpu_manager.go:414] "containerMap: removing container" podUID="c9fa2248-8e6f-413b-858b-ca0b9a363765" containerName="maas-api" Oct 30 18:40:09.429507 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:40:09.429469 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9fa2248-8e6f-413b-858b-ca0b9a363765" containerName="maas-api" Oct 30 18:40:09.429635 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:40:09.429546 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9fa2248-8e6f-413b-858b-ca0b9a363765" containerName="maas-api" Oct 30 18:40:09.429635 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:40:09.429555 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fff7b54-a723-42aa-ae6f-d5a3db247885" containerName="authorino" Oct 30 18:40:09.433187 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:40:09.433171 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-xkxkl" Oct 30 18:40:09.442497 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:40:09.442467 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-xkxkl"] Oct 30 18:40:09.500629 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:40:09.500594 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7tfc\" (UniqueName: \"kubernetes.io/projected/13296062-8ceb-4447-9418-d52de1a4b712-kube-api-access-t7tfc\") pod \"llmisvc-controller-manager-6466c554d8-xkxkl\" (UID: \"13296062-8ceb-4447-9418-d52de1a4b712\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-xkxkl" Oct 30 18:40:09.500768 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:40:09.500705 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/13296062-8ceb-4447-9418-d52de1a4b712-cert\") pod \"llmisvc-controller-manager-6466c554d8-xkxkl\" (UID: \"13296062-8ceb-4447-9418-d52de1a4b712\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-xkxkl" Oct 30 18:40:09.601559 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:40:09.601530 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/13296062-8ceb-4447-9418-d52de1a4b712-cert\") pod \"llmisvc-controller-manager-6466c554d8-xkxkl\" (UID: \"13296062-8ceb-4447-9418-d52de1a4b712\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-xkxkl" Oct 30 18:40:09.601695 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:40:09.601599 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7tfc\" (UniqueName: \"kubernetes.io/projected/13296062-8ceb-4447-9418-d52de1a4b712-kube-api-access-t7tfc\") pod \"llmisvc-controller-manager-6466c554d8-xkxkl\" (UID: \"13296062-8ceb-4447-9418-d52de1a4b712\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-xkxkl" Oct 30 18:40:09.604160 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:40:09.604123 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/13296062-8ceb-4447-9418-d52de1a4b712-cert\") pod \"llmisvc-controller-manager-6466c554d8-xkxkl\" (UID: \"13296062-8ceb-4447-9418-d52de1a4b712\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-xkxkl" Oct 30 18:40:09.613896 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:40:09.613874 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7tfc\" (UniqueName: \"kubernetes.io/projected/13296062-8ceb-4447-9418-d52de1a4b712-kube-api-access-t7tfc\") pod \"llmisvc-controller-manager-6466c554d8-xkxkl\" (UID: \"13296062-8ceb-4447-9418-d52de1a4b712\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-xkxkl" Oct 30 18:40:09.746395 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:40:09.746315 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-xkxkl" Oct 30 18:40:10.016389 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:40:10.016311 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-xkxkl"] Oct 30 18:40:10.019346 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:40:10.019310 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13296062_8ceb_4447_9418_d52de1a4b712.slice/crio-ea31d224ecef4d8e28472c4a092f052e6eca07403cbec5d1735d6d62a960d78a WatchSource:0}: Error finding container ea31d224ecef4d8e28472c4a092f052e6eca07403cbec5d1735d6d62a960d78a: Status 404 returned error can't find the container with id ea31d224ecef4d8e28472c4a092f052e6eca07403cbec5d1735d6d62a960d78a Oct 30 18:40:10.317678 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:40:10.317643 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-xkxkl" event={"ID":"13296062-8ceb-4447-9418-d52de1a4b712","Type":"ContainerStarted","Data":"ea31d224ecef4d8e28472c4a092f052e6eca07403cbec5d1735d6d62a960d78a"} Oct 30 18:40:11.323795 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:40:11.323757 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-xkxkl" event={"ID":"13296062-8ceb-4447-9418-d52de1a4b712","Type":"ContainerStarted","Data":"75c76e275aa9b41cd65d106581febe49c2da563f0507f5baea5ba85980b2c14f"} Oct 30 18:40:11.324238 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:40:11.323891 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-xkxkl" Oct 30 18:40:11.343299 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:40:11.343255 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-xkxkl" podStartSLOduration=1.9796104350000001 podStartE2EDuration="2.343240883s" totalImagesPullingTime="363.630448ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:40:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:40:11.341607177 +0000 UTC m=+378.870638316" watchObservedRunningTime="2026-10-30 18:40:11.343240883 +0000 UTC m=+378.872272020" Oct 30 18:40:23.930746 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:40:23.930711 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 18:40:42.330538 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:40:42.330507 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-xkxkl" Oct 30 18:40:42.379064 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:40:42.379033 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-hwhlk"] Oct 30 18:40:42.379453 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:40:42.379286 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-hwhlk" podUID="37cb80da-410f-4ff5-98e2-34eae144104f" containerName="manager" containerID="cri-o://e91549592797bdfd41dbe95a747c11d0d411a0688e1857468e58720da7d41e5d" gracePeriod=30 Oct 30 18:40:42.634345 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:40:42.634319 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-hwhlk" Oct 30 18:40:42.806748 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:40:42.806717 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37cb80da-410f-4ff5-98e2-34eae144104f-cert\" (UniqueName: \"kubernetes.io/secret/37cb80da-410f-4ff5-98e2-34eae144104f-cert\") pod \"37cb80da-410f-4ff5-98e2-34eae144104f\" (UID: \"37cb80da-410f-4ff5-98e2-34eae144104f\") " Oct 30 18:40:42.806931 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:40:42.806789 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37cb80da-410f-4ff5-98e2-34eae144104f-kube-api-access-qgpv6\" (UniqueName: \"kubernetes.io/projected/37cb80da-410f-4ff5-98e2-34eae144104f-kube-api-access-qgpv6\") pod \"37cb80da-410f-4ff5-98e2-34eae144104f\" (UID: \"37cb80da-410f-4ff5-98e2-34eae144104f\") " Oct 30 18:40:42.809049 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:40:42.809023 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37cb80da-410f-4ff5-98e2-34eae144104f-kube-api-access-qgpv6" pod "37cb80da-410f-4ff5-98e2-34eae144104f" (UID: "37cb80da-410f-4ff5-98e2-34eae144104f"). InnerVolumeSpecName "kube-api-access-qgpv6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:40:42.809049 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:40:42.809028 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37cb80da-410f-4ff5-98e2-34eae144104f-cert" pod "37cb80da-410f-4ff5-98e2-34eae144104f" (UID: "37cb80da-410f-4ff5-98e2-34eae144104f"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:40:42.908127 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:40:42.908051 6960 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/37cb80da-410f-4ff5-98e2-34eae144104f-cert\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:40:42.908127 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:40:42.908076 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgpv6\" (UniqueName: \"kubernetes.io/projected/37cb80da-410f-4ff5-98e2-34eae144104f-kube-api-access-qgpv6\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:40:43.429482 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:40:43.429449 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:43.471022 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:40:43.470989 6960 generic.go:356] "Generic (PLEG): container finished" podID="37cb80da-410f-4ff5-98e2-34eae144104f" containerID="e91549592797bdfd41dbe95a747c11d0d411a0688e1857468e58720da7d41e5d" exitCode=0 Oct 30 18:40:43.471205 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:40:43.471035 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-hwhlk" Oct 30 18:40:43.471205 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:40:43.471040 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-hwhlk" event={"ID":"37cb80da-410f-4ff5-98e2-34eae144104f","Type":"ContainerDied","Data":"e91549592797bdfd41dbe95a747c11d0d411a0688e1857468e58720da7d41e5d"} Oct 30 18:40:43.471205 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:40:43.471090 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-hwhlk" event={"ID":"37cb80da-410f-4ff5-98e2-34eae144104f","Type":"ContainerDied","Data":"ed339f319a466c92e4a6fea626c5d29ad733aed78623ba7557bfdf684a16cd7a"} Oct 30 18:40:43.471205 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:40:43.471119 6960 scope.go:122] "RemoveContainer" containerID="e91549592797bdfd41dbe95a747c11d0d411a0688e1857468e58720da7d41e5d" Oct 30 18:40:43.480129 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:40:43.480114 6960 scope.go:122] "RemoveContainer" containerID="e91549592797bdfd41dbe95a747c11d0d411a0688e1857468e58720da7d41e5d" Oct 30 18:40:43.480401 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:40:43.480379 6960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e91549592797bdfd41dbe95a747c11d0d411a0688e1857468e58720da7d41e5d\": container with ID starting with e91549592797bdfd41dbe95a747c11d0d411a0688e1857468e58720da7d41e5d not found: ID does not exist" containerID="e91549592797bdfd41dbe95a747c11d0d411a0688e1857468e58720da7d41e5d" Oct 30 18:40:43.480491 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:40:43.480408 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e91549592797bdfd41dbe95a747c11d0d411a0688e1857468e58720da7d41e5d"} err="failed to get container status \"e91549592797bdfd41dbe95a747c11d0d411a0688e1857468e58720da7d41e5d\": rpc error: code = NotFound desc = could not find container \"e91549592797bdfd41dbe95a747c11d0d411a0688e1857468e58720da7d41e5d\": container with ID starting with e91549592797bdfd41dbe95a747c11d0d411a0688e1857468e58720da7d41e5d not found: ID does not exist" Oct 30 18:40:43.491588 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:40:43.491567 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-hwhlk"] Oct 30 18:40:43.499027 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:40:43.499007 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-hwhlk"] Oct 30 18:40:44.591726 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:40:44.591691 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37cb80da-410f-4ff5-98e2-34eae144104f" path="/var/lib/kubelet/pods/37cb80da-410f-4ff5-98e2-34eae144104f/volumes" Oct 30 18:40:45.135931 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:40:45.135900 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:45.252476 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:40:45.252442 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:49.796874 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:40:49.796839 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:53.931588 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:40:53.931546 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 18:41:01.922033 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:41:01.921998 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:03.290440 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:41:03.290403 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:06.892081 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:41:06.892044 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:07.060833 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:41:07.060802 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:23.931886 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:41:23.931844 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 18:41:31.554374 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:41:31.554335 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:32.093895 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:41:32.093864 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:33.424539 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:41:33.424457 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:33.533080 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:41:33.533040 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:38.148385 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:41:38.148345 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:40.121092 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:41:40.121056 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:40.995826 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:41:40.995793 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:41.985799 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:41:41.985751 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:44.866564 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:41:44.866525 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:45.829252 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:41:45.829218 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:47.896526 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:41:47.896479 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:49.049069 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:41:49.049032 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:49.908906 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:41:49.908872 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:53.930874 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:41:53.930836 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 18:42:06.076483 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:06.076443 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:06.116683 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:06.116654 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pb8bs"] Oct 30 18:42:06.117082 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:06.117070 6960 cpu_manager.go:414] "containerMap: removing container" podUID="37cb80da-410f-4ff5-98e2-34eae144104f" containerName="manager" Oct 30 18:42:06.117124 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:06.117084 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37cb80da-410f-4ff5-98e2-34eae144104f" containerName="manager" Oct 30 18:42:06.117183 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:06.117173 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="37cb80da-410f-4ff5-98e2-34eae144104f" containerName="manager" Oct 30 18:42:06.120020 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:06.120000 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pb8bs" Oct 30 18:42:06.128633 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:06.128614 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-n2l29\"" Oct 30 18:42:06.144943 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:06.144921 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pb8bs"] Oct 30 18:42:06.223747 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:06.223717 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:06.286316 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:06.286284 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/830b7d8c-c70c-4242-b81a-8be190df4aa1-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-pb8bs\" (UID: \"830b7d8c-c70c-4242-b81a-8be190df4aa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pb8bs" Oct 30 18:42:06.286316 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:06.286318 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/830b7d8c-c70c-4242-b81a-8be190df4aa1-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-pb8bs\" (UID: \"830b7d8c-c70c-4242-b81a-8be190df4aa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pb8bs" Oct 30 18:42:06.286529 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:06.286340 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgt44\" (UniqueName: \"kubernetes.io/projected/830b7d8c-c70c-4242-b81a-8be190df4aa1-kube-api-access-wgt44\") pod \"maas-default-gateway-openshift-default-855f9574cf-pb8bs\" (UID: \"830b7d8c-c70c-4242-b81a-8be190df4aa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pb8bs" Oct 30 18:42:06.286529 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:06.286381 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/830b7d8c-c70c-4242-b81a-8be190df4aa1-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-pb8bs\" (UID: \"830b7d8c-c70c-4242-b81a-8be190df4aa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pb8bs" Oct 30 18:42:06.286529 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:06.286469 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/830b7d8c-c70c-4242-b81a-8be190df4aa1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-pb8bs\" (UID: \"830b7d8c-c70c-4242-b81a-8be190df4aa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pb8bs" Oct 30 18:42:06.286529 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:06.286486 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/830b7d8c-c70c-4242-b81a-8be190df4aa1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-pb8bs\" (UID: \"830b7d8c-c70c-4242-b81a-8be190df4aa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pb8bs" Oct 30 18:42:06.286529 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:06.286502 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/830b7d8c-c70c-4242-b81a-8be190df4aa1-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-pb8bs\" (UID: \"830b7d8c-c70c-4242-b81a-8be190df4aa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pb8bs" Oct 30 18:42:06.286685 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:06.286532 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/830b7d8c-c70c-4242-b81a-8be190df4aa1-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-pb8bs\" (UID: \"830b7d8c-c70c-4242-b81a-8be190df4aa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pb8bs" Oct 30 18:42:06.286718 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:06.286680 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/830b7d8c-c70c-4242-b81a-8be190df4aa1-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-pb8bs\" (UID: \"830b7d8c-c70c-4242-b81a-8be190df4aa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pb8bs" Oct 30 18:42:06.387384 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:06.387300 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/830b7d8c-c70c-4242-b81a-8be190df4aa1-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-pb8bs\" (UID: \"830b7d8c-c70c-4242-b81a-8be190df4aa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pb8bs" Oct 30 18:42:06.387384 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:06.387360 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/830b7d8c-c70c-4242-b81a-8be190df4aa1-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-pb8bs\" (UID: \"830b7d8c-c70c-4242-b81a-8be190df4aa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pb8bs" Oct 30 18:42:06.387604 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:06.387395 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/830b7d8c-c70c-4242-b81a-8be190df4aa1-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-pb8bs\" (UID: \"830b7d8c-c70c-4242-b81a-8be190df4aa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pb8bs" Oct 30 18:42:06.387604 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:06.387411 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/830b7d8c-c70c-4242-b81a-8be190df4aa1-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-pb8bs\" (UID: \"830b7d8c-c70c-4242-b81a-8be190df4aa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pb8bs" Oct 30 18:42:06.387604 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:06.387437 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgt44\" (UniqueName: \"kubernetes.io/projected/830b7d8c-c70c-4242-b81a-8be190df4aa1-kube-api-access-wgt44\") pod \"maas-default-gateway-openshift-default-855f9574cf-pb8bs\" (UID: \"830b7d8c-c70c-4242-b81a-8be190df4aa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pb8bs" Oct 30 18:42:06.387604 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:06.387466 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/830b7d8c-c70c-4242-b81a-8be190df4aa1-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-pb8bs\" (UID: \"830b7d8c-c70c-4242-b81a-8be190df4aa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pb8bs" Oct 30 18:42:06.387604 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:06.387565 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/830b7d8c-c70c-4242-b81a-8be190df4aa1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-pb8bs\" (UID: \"830b7d8c-c70c-4242-b81a-8be190df4aa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pb8bs" Oct 30 18:42:06.387604 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:06.387589 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/830b7d8c-c70c-4242-b81a-8be190df4aa1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-pb8bs\" (UID: \"830b7d8c-c70c-4242-b81a-8be190df4aa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pb8bs" Oct 30 18:42:06.387910 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:06.387614 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/830b7d8c-c70c-4242-b81a-8be190df4aa1-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-pb8bs\" (UID: \"830b7d8c-c70c-4242-b81a-8be190df4aa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pb8bs" Oct 30 18:42:06.387910 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:06.387751 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/830b7d8c-c70c-4242-b81a-8be190df4aa1-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-pb8bs\" (UID: \"830b7d8c-c70c-4242-b81a-8be190df4aa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pb8bs" Oct 30 18:42:06.387910 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:06.387775 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/830b7d8c-c70c-4242-b81a-8be190df4aa1-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-pb8bs\" (UID: \"830b7d8c-c70c-4242-b81a-8be190df4aa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pb8bs" Oct 30 18:42:06.388069 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:06.387976 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/830b7d8c-c70c-4242-b81a-8be190df4aa1-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-pb8bs\" (UID: \"830b7d8c-c70c-4242-b81a-8be190df4aa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pb8bs" Oct 30 18:42:06.388069 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:06.387990 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/830b7d8c-c70c-4242-b81a-8be190df4aa1-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-pb8bs\" (UID: \"830b7d8c-c70c-4242-b81a-8be190df4aa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pb8bs" Oct 30 18:42:06.388322 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:06.388302 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/830b7d8c-c70c-4242-b81a-8be190df4aa1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-pb8bs\" (UID: \"830b7d8c-c70c-4242-b81a-8be190df4aa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pb8bs" Oct 30 18:42:06.389960 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:06.389924 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/830b7d8c-c70c-4242-b81a-8be190df4aa1-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-pb8bs\" (UID: \"830b7d8c-c70c-4242-b81a-8be190df4aa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pb8bs" Oct 30 18:42:06.390376 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:06.390352 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/830b7d8c-c70c-4242-b81a-8be190df4aa1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-pb8bs\" (UID: \"830b7d8c-c70c-4242-b81a-8be190df4aa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pb8bs" Oct 30 18:42:06.402376 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:06.402349 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgt44\" (UniqueName: \"kubernetes.io/projected/830b7d8c-c70c-4242-b81a-8be190df4aa1-kube-api-access-wgt44\") pod \"maas-default-gateway-openshift-default-855f9574cf-pb8bs\" (UID: \"830b7d8c-c70c-4242-b81a-8be190df4aa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pb8bs" Oct 30 18:42:06.402610 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:06.402593 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/830b7d8c-c70c-4242-b81a-8be190df4aa1-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-pb8bs\" (UID: \"830b7d8c-c70c-4242-b81a-8be190df4aa1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pb8bs" Oct 30 18:42:06.433655 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:06.433629 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pb8bs" Oct 30 18:42:06.734820 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:06.734731 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pb8bs"] Oct 30 18:42:06.737695 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:42:06.737670 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod830b7d8c_c70c_4242_b81a_8be190df4aa1.slice/crio-945789fdf6a8446db684fe5e517fcf94845a107e7757fa5bd6ed11cfb62e2db0 WatchSource:0}: Error finding container 945789fdf6a8446db684fe5e517fcf94845a107e7757fa5bd6ed11cfb62e2db0: Status 404 returned error can't find the container with id 945789fdf6a8446db684fe5e517fcf94845a107e7757fa5bd6ed11cfb62e2db0 Oct 30 18:42:06.739822 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:06.739765 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:42:06.739895 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:06.739855 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:42:06.739895 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:06.739885 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:42:06.861734 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:06.861697 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pb8bs" event={"ID":"830b7d8c-c70c-4242-b81a-8be190df4aa1","Type":"ContainerStarted","Data":"8a884d0b39a6f7dbad9c234ccc97e1353f4933ead86878f61a364bb680e88c0b"} Oct 30 18:42:06.861892 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:06.861753 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pb8bs" event={"ID":"830b7d8c-c70c-4242-b81a-8be190df4aa1","Type":"ContainerStarted","Data":"945789fdf6a8446db684fe5e517fcf94845a107e7757fa5bd6ed11cfb62e2db0"} Oct 30 18:42:06.890626 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:06.890573 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pb8bs" podStartSLOduration=0.890558375 podStartE2EDuration="890.558375ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:42:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:42:06.88890289 +0000 UTC m=+494.417934027" watchObservedRunningTime="2026-10-30 18:42:06.890558375 +0000 UTC m=+494.419589512" Oct 30 18:42:07.434855 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:07.434821 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pb8bs" Oct 30 18:42:07.436377 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:07.436353 6960 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-pb8bs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 30 18:42:07.436487 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:07.436400 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pb8bs" podUID="830b7d8c-c70c-4242-b81a-8be190df4aa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 30 18:42:07.708852 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:07.708759 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:07.833516 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:07.833479 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:08.434847 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:08.434819 6960 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-pb8bs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 30 18:42:08.435030 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:08.434870 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pb8bs" podUID="830b7d8c-c70c-4242-b81a-8be190df4aa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 30 18:42:09.434312 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:09.434275 6960 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-pb8bs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 30 18:42:09.434490 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:09.434325 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pb8bs" podUID="830b7d8c-c70c-4242-b81a-8be190df4aa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 30 18:42:10.434248 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:10.434214 6960 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-pb8bs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 30 18:42:10.434633 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:10.434269 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pb8bs" podUID="830b7d8c-c70c-4242-b81a-8be190df4aa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 30 18:42:11.434562 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:11.434528 6960 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-pb8bs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 30 18:42:11.435060 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:11.434595 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pb8bs" podUID="830b7d8c-c70c-4242-b81a-8be190df4aa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 30 18:42:12.434403 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:12.434369 6960 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-pb8bs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 30 18:42:12.434606 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:12.434448 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pb8bs" podUID="830b7d8c-c70c-4242-b81a-8be190df4aa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 30 18:42:13.434821 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:13.434773 6960 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-pb8bs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 30 18:42:13.435381 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:13.434829 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pb8bs" podUID="830b7d8c-c70c-4242-b81a-8be190df4aa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 30 18:42:14.957838 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:14.957803 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pb8bs" Oct 30 18:42:14.958818 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:14.958788 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pb8bs" Oct 30 18:42:14.961566 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:14.959945 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pb8bs" Oct 30 18:42:21.225195 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:21.225157 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d58l4"] Oct 30 18:42:21.228550 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:21.228528 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d58l4" Oct 30 18:42:21.256317 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:21.256288 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d58l4"] Oct 30 18:42:21.323513 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:21.323480 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42b9fddf-ca5b-4181-8819-c32996d6ff0e-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-d58l4\" (UID: \"42b9fddf-ca5b-4181-8819-c32996d6ff0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d58l4" Oct 30 18:42:21.323680 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:21.323539 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42b9fddf-ca5b-4181-8819-c32996d6ff0e-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-d58l4\" (UID: \"42b9fddf-ca5b-4181-8819-c32996d6ff0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d58l4" Oct 30 18:42:21.323680 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:21.323644 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42b9fddf-ca5b-4181-8819-c32996d6ff0e-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-d58l4\" (UID: \"42b9fddf-ca5b-4181-8819-c32996d6ff0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d58l4" Oct 30 18:42:21.323762 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:21.323706 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42b9fddf-ca5b-4181-8819-c32996d6ff0e-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-d58l4\" (UID: \"42b9fddf-ca5b-4181-8819-c32996d6ff0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d58l4" Oct 30 18:42:21.323840 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:21.323821 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42b9fddf-ca5b-4181-8819-c32996d6ff0e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-d58l4\" (UID: \"42b9fddf-ca5b-4181-8819-c32996d6ff0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d58l4" Oct 30 18:42:21.323891 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:21.323872 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42b9fddf-ca5b-4181-8819-c32996d6ff0e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-d58l4\" (UID: \"42b9fddf-ca5b-4181-8819-c32996d6ff0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d58l4" Oct 30 18:42:21.323946 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:21.323930 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42b9fddf-ca5b-4181-8819-c32996d6ff0e-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-d58l4\" (UID: \"42b9fddf-ca5b-4181-8819-c32996d6ff0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d58l4" Oct 30 18:42:21.323994 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:21.323982 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42b9fddf-ca5b-4181-8819-c32996d6ff0e-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-d58l4\" (UID: \"42b9fddf-ca5b-4181-8819-c32996d6ff0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d58l4" Oct 30 18:42:21.324033 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:21.324006 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mcwp\" (UniqueName: \"kubernetes.io/projected/42b9fddf-ca5b-4181-8819-c32996d6ff0e-kube-api-access-8mcwp\") pod \"maas-default-gateway-openshift-default-855f9574cf-d58l4\" (UID: \"42b9fddf-ca5b-4181-8819-c32996d6ff0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d58l4" Oct 30 18:42:21.425587 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:21.425549 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42b9fddf-ca5b-4181-8819-c32996d6ff0e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-d58l4\" (UID: \"42b9fddf-ca5b-4181-8819-c32996d6ff0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d58l4" Oct 30 18:42:21.425787 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:21.425600 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42b9fddf-ca5b-4181-8819-c32996d6ff0e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-d58l4\" (UID: \"42b9fddf-ca5b-4181-8819-c32996d6ff0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d58l4" Oct 30 18:42:21.425787 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:21.425617 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42b9fddf-ca5b-4181-8819-c32996d6ff0e-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-d58l4\" (UID: \"42b9fddf-ca5b-4181-8819-c32996d6ff0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d58l4" Oct 30 18:42:21.425787 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:21.425641 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42b9fddf-ca5b-4181-8819-c32996d6ff0e-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-d58l4\" (UID: \"42b9fddf-ca5b-4181-8819-c32996d6ff0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d58l4" Oct 30 18:42:21.425787 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:21.425659 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mcwp\" (UniqueName: \"kubernetes.io/projected/42b9fddf-ca5b-4181-8819-c32996d6ff0e-kube-api-access-8mcwp\") pod \"maas-default-gateway-openshift-default-855f9574cf-d58l4\" (UID: \"42b9fddf-ca5b-4181-8819-c32996d6ff0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d58l4" Oct 30 18:42:21.425787 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:21.425686 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42b9fddf-ca5b-4181-8819-c32996d6ff0e-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-d58l4\" (UID: \"42b9fddf-ca5b-4181-8819-c32996d6ff0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d58l4" Oct 30 18:42:21.425787 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:21.425722 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42b9fddf-ca5b-4181-8819-c32996d6ff0e-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-d58l4\" (UID: \"42b9fddf-ca5b-4181-8819-c32996d6ff0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d58l4" Oct 30 18:42:21.425787 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:21.425756 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42b9fddf-ca5b-4181-8819-c32996d6ff0e-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-d58l4\" (UID: \"42b9fddf-ca5b-4181-8819-c32996d6ff0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d58l4" Oct 30 18:42:21.426200 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:21.425794 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42b9fddf-ca5b-4181-8819-c32996d6ff0e-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-d58l4\" (UID: \"42b9fddf-ca5b-4181-8819-c32996d6ff0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d58l4" Oct 30 18:42:21.426325 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:21.426298 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42b9fddf-ca5b-4181-8819-c32996d6ff0e-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-d58l4\" (UID: \"42b9fddf-ca5b-4181-8819-c32996d6ff0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d58l4" Oct 30 18:42:21.426393 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:21.426347 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42b9fddf-ca5b-4181-8819-c32996d6ff0e-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-d58l4\" (UID: \"42b9fddf-ca5b-4181-8819-c32996d6ff0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d58l4" Oct 30 18:42:21.426450 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:21.426411 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42b9fddf-ca5b-4181-8819-c32996d6ff0e-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-d58l4\" (UID: \"42b9fddf-ca5b-4181-8819-c32996d6ff0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d58l4" Oct 30 18:42:21.426450 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:21.426424 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42b9fddf-ca5b-4181-8819-c32996d6ff0e-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-d58l4\" (UID: \"42b9fddf-ca5b-4181-8819-c32996d6ff0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d58l4" Oct 30 18:42:21.426450 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:21.426428 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42b9fddf-ca5b-4181-8819-c32996d6ff0e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-d58l4\" (UID: \"42b9fddf-ca5b-4181-8819-c32996d6ff0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d58l4" Oct 30 18:42:21.428214 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:21.428197 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42b9fddf-ca5b-4181-8819-c32996d6ff0e-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-d58l4\" (UID: \"42b9fddf-ca5b-4181-8819-c32996d6ff0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d58l4" Oct 30 18:42:21.428473 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:21.428457 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42b9fddf-ca5b-4181-8819-c32996d6ff0e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-d58l4\" (UID: \"42b9fddf-ca5b-4181-8819-c32996d6ff0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d58l4" Oct 30 18:42:21.438328 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:21.438304 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mcwp\" (UniqueName: \"kubernetes.io/projected/42b9fddf-ca5b-4181-8819-c32996d6ff0e-kube-api-access-8mcwp\") pod \"maas-default-gateway-openshift-default-855f9574cf-d58l4\" (UID: \"42b9fddf-ca5b-4181-8819-c32996d6ff0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d58l4" Oct 30 18:42:21.442576 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:21.442548 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42b9fddf-ca5b-4181-8819-c32996d6ff0e-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-d58l4\" (UID: \"42b9fddf-ca5b-4181-8819-c32996d6ff0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d58l4" Oct 30 18:42:21.542134 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:21.542048 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d58l4" Oct 30 18:42:21.824612 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:42:21.824586 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42b9fddf_ca5b_4181_8819_c32996d6ff0e.slice/crio-c85ab110636348dd86f24e56d79b459eb42ffc5b10de7aafc639e50fb4efe7ca WatchSource:0}: Error finding container c85ab110636348dd86f24e56d79b459eb42ffc5b10de7aafc639e50fb4efe7ca: Status 404 returned error can't find the container with id c85ab110636348dd86f24e56d79b459eb42ffc5b10de7aafc639e50fb4efe7ca Oct 30 18:42:21.824736 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:21.824716 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d58l4"] Oct 30 18:42:21.826847 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:21.826820 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:42:21.826921 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:21.826880 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:42:21.826921 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:21.826913 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:42:21.945944 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:21.945913 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d58l4" event={"ID":"42b9fddf-ca5b-4181-8819-c32996d6ff0e","Type":"ContainerStarted","Data":"4de5bb8b562e6da61eaddd7604fb2e7e810953eaee3bed6dacf67fb993737fb6"} Oct 30 18:42:21.946131 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:21.945969 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d58l4" event={"ID":"42b9fddf-ca5b-4181-8819-c32996d6ff0e","Type":"ContainerStarted","Data":"c85ab110636348dd86f24e56d79b459eb42ffc5b10de7aafc639e50fb4efe7ca"} Oct 30 18:42:21.972882 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:21.972680 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d58l4" podStartSLOduration=0.972646975 podStartE2EDuration="972.646975ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:42:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:42:21.972103749 +0000 UTC m=+509.501134887" watchObservedRunningTime="2026-10-30 18:42:21.972646975 +0000 UTC m=+509.501678113" Oct 30 18:42:22.542256 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:22.542223 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d58l4" Oct 30 18:42:22.543471 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:22.543450 6960 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-d58l4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 18:42:22.543536 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:22.543497 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d58l4" podUID="42b9fddf-ca5b-4181-8819-c32996d6ff0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 18:42:22.878169 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:22.875652 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:22.996425 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:22.996397 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:23.543412 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:23.543375 6960 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-d58l4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 18:42:23.543809 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:23.543432 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d58l4" podUID="42b9fddf-ca5b-4181-8819-c32996d6ff0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 18:42:23.931263 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:42:23.931232 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 18:42:24.542982 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:24.542949 6960 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-d58l4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 18:42:24.543178 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:24.543001 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d58l4" podUID="42b9fddf-ca5b-4181-8819-c32996d6ff0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 18:42:25.543445 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:25.543413 6960 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-d58l4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 18:42:25.543836 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:25.543465 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d58l4" podUID="42b9fddf-ca5b-4181-8819-c32996d6ff0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 18:42:26.542605 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:26.542572 6960 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-d58l4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 18:42:26.542790 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:26.542622 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d58l4" podUID="42b9fddf-ca5b-4181-8819-c32996d6ff0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 18:42:27.543067 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:27.543028 6960 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-d58l4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 18:42:27.543479 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:27.543089 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d58l4" podUID="42b9fddf-ca5b-4181-8819-c32996d6ff0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 18:42:28.542996 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:28.542954 6960 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-d58l4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 18:42:28.543220 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:28.543079 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d58l4" podUID="42b9fddf-ca5b-4181-8819-c32996d6ff0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 18:42:29.930553 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:29.930521 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d58l4" Oct 30 18:42:29.931083 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:29.931060 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d58l4" Oct 30 18:42:29.932602 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:29.931896 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d58l4" Oct 30 18:42:34.686322 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:34.686281 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:34.817576 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:34.817539 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:34.948852 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:34.948772 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:35.217875 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:35.217787 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:35.362072 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:35.362039 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:35.525366 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:35.525210 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:35.862486 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:35.862454 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:36.100516 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:36.100478 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:36.844810 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:36.844776 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:37.348208 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:37.348174 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:37.632216 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:37.629875 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:37.894022 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:37.893932 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:38.099012 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:38.098981 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:38.274076 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:38.273972 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:39.153962 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:39.153934 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:39.386806 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:39.386777 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:39.776847 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:39.776809 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:39.965356 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:39.965316 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:40.115428 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:40.115392 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:40.260479 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:40.260440 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:40.390110 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:40.390030 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:40.481092 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:40.481060 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f5768279-5cfb5444bd-mdhlb"] Oct 30 18:42:40.485227 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:40.485206 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f5768279-5cfb5444bd-mdhlb" Oct 30 18:42:40.500523 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:40.500494 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f5768279-5cfb5444bd-mdhlb"] Oct 30 18:42:40.557909 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:40.557877 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:40.565977 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:40.565926 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-plgsl\"" Oct 30 18:42:40.566124 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:40.566051 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 18:42:40.566124 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:40.566072 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-f5768279-serving-cert\"" Oct 30 18:42:40.567890 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:40.567877 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:42:40.567934 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:40.567897 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:42:40.635996 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:40.635964 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac740be5-4f91-4988-8321-b73ea6d7a842-openshift-service-ca\") pod \"maas-api-e2e-ait-f5768279-5cfb5444bd-mdhlb\" (UID: \"ac740be5-4f91-4988-8321-b73ea6d7a842\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f5768279-5cfb5444bd-mdhlb" Oct 30 18:42:40.636258 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:40.636041 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5hxm\" (UniqueName: \"kubernetes.io/projected/ac740be5-4f91-4988-8321-b73ea6d7a842-kube-api-access-g5hxm\") pod \"maas-api-e2e-ait-f5768279-5cfb5444bd-mdhlb\" (UID: \"ac740be5-4f91-4988-8321-b73ea6d7a842\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f5768279-5cfb5444bd-mdhlb" Oct 30 18:42:40.636258 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:40.636245 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac740be5-4f91-4988-8321-b73ea6d7a842-maas-api-tls\") pod \"maas-api-e2e-ait-f5768279-5cfb5444bd-mdhlb\" (UID: \"ac740be5-4f91-4988-8321-b73ea6d7a842\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f5768279-5cfb5444bd-mdhlb" Oct 30 18:42:40.636402 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:40.636361 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac740be5-4f91-4988-8321-b73ea6d7a842-metrics-certs\") pod \"maas-api-e2e-ait-f5768279-5cfb5444bd-mdhlb\" (UID: \"ac740be5-4f91-4988-8321-b73ea6d7a842\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f5768279-5cfb5444bd-mdhlb" Oct 30 18:42:40.737103 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:40.737035 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac740be5-4f91-4988-8321-b73ea6d7a842-openshift-service-ca\") pod \"maas-api-e2e-ait-f5768279-5cfb5444bd-mdhlb\" (UID: \"ac740be5-4f91-4988-8321-b73ea6d7a842\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f5768279-5cfb5444bd-mdhlb" Oct 30 18:42:40.737349 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:40.737331 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5hxm\" (UniqueName: \"kubernetes.io/projected/ac740be5-4f91-4988-8321-b73ea6d7a842-kube-api-access-g5hxm\") pod \"maas-api-e2e-ait-f5768279-5cfb5444bd-mdhlb\" (UID: \"ac740be5-4f91-4988-8321-b73ea6d7a842\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f5768279-5cfb5444bd-mdhlb" Oct 30 18:42:40.737510 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:40.737488 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac740be5-4f91-4988-8321-b73ea6d7a842-maas-api-tls\") pod \"maas-api-e2e-ait-f5768279-5cfb5444bd-mdhlb\" (UID: \"ac740be5-4f91-4988-8321-b73ea6d7a842\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f5768279-5cfb5444bd-mdhlb" Oct 30 18:42:40.737588 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:40.737555 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac740be5-4f91-4988-8321-b73ea6d7a842-metrics-certs\") pod \"maas-api-e2e-ait-f5768279-5cfb5444bd-mdhlb\" (UID: \"ac740be5-4f91-4988-8321-b73ea6d7a842\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f5768279-5cfb5444bd-mdhlb" Oct 30 18:42:40.737796 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:40.737773 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac740be5-4f91-4988-8321-b73ea6d7a842-openshift-service-ca\") pod \"maas-api-e2e-ait-f5768279-5cfb5444bd-mdhlb\" (UID: \"ac740be5-4f91-4988-8321-b73ea6d7a842\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f5768279-5cfb5444bd-mdhlb" Oct 30 18:42:40.740215 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:40.740191 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac740be5-4f91-4988-8321-b73ea6d7a842-maas-api-tls\") pod \"maas-api-e2e-ait-f5768279-5cfb5444bd-mdhlb\" (UID: \"ac740be5-4f91-4988-8321-b73ea6d7a842\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f5768279-5cfb5444bd-mdhlb" Oct 30 18:42:40.740307 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:40.740221 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac740be5-4f91-4988-8321-b73ea6d7a842-metrics-certs\") pod \"maas-api-e2e-ait-f5768279-5cfb5444bd-mdhlb\" (UID: \"ac740be5-4f91-4988-8321-b73ea6d7a842\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f5768279-5cfb5444bd-mdhlb" Oct 30 18:42:40.759947 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:40.759925 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5hxm\" (UniqueName: \"kubernetes.io/projected/ac740be5-4f91-4988-8321-b73ea6d7a842-kube-api-access-g5hxm\") pod \"maas-api-e2e-ait-f5768279-5cfb5444bd-mdhlb\" (UID: \"ac740be5-4f91-4988-8321-b73ea6d7a842\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f5768279-5cfb5444bd-mdhlb" Oct 30 18:42:40.799841 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:40.799803 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f5768279-5cfb5444bd-mdhlb" Oct 30 18:42:40.824578 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:40.824553 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:41.094514 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:41.093087 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f5768279-5cfb5444bd-mdhlb"] Oct 30 18:42:41.099297 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:42:41.099266 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac740be5_4f91_4988_8321_b73ea6d7a842.slice/crio-7bd811383032b0332e343918d7ea98f878cc07366cf0ad29c343ea873358de12 WatchSource:0}: Error finding container 7bd811383032b0332e343918d7ea98f878cc07366cf0ad29c343ea873358de12: Status 404 returned error can't find the container with id 7bd811383032b0332e343918d7ea98f878cc07366cf0ad29c343ea873358de12 Oct 30 18:42:41.752187 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:41.752154 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:41.868802 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:41.868728 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:41.983383 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:41.983353 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:42.067734 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:42.067699 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f5768279-5cfb5444bd-mdhlb" event={"ID":"ac740be5-4f91-4988-8321-b73ea6d7a842","Type":"ContainerStarted","Data":"71e81e6a74177e98a180b2b44760f0aa7e614f36f9a11cedf7fc9096182e62ec"} Oct 30 18:42:42.067899 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:42.067760 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f5768279-5cfb5444bd-mdhlb" event={"ID":"ac740be5-4f91-4988-8321-b73ea6d7a842","Type":"ContainerStarted","Data":"7bd811383032b0332e343918d7ea98f878cc07366cf0ad29c343ea873358de12"} Oct 30 18:42:42.067899 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:42.067833 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f5768279-5cfb5444bd-mdhlb" Oct 30 18:42:42.092130 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:42.092090 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f5768279-5cfb5444bd-mdhlb" podStartSLOduration=1.617793484 podStartE2EDuration="2.092077049s" totalImagesPullingTime="474.283565ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:42:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:42:42.090169465 +0000 UTC m=+529.619200594" watchObservedRunningTime="2026-10-30 18:42:42.092077049 +0000 UTC m=+529.621108530" Oct 30 18:42:42.112593 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:42.112570 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:42.197130 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:42.197066 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9aded813-86dd685f96-5fk8w"] Oct 30 18:42:42.200498 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:42.200478 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9aded813-86dd685f96-5fk8w" Oct 30 18:42:42.208581 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:42.208561 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-9aded813-dockercfg-7xmlt\"" Oct 30 18:42:42.213664 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:42.213642 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9aded813-86dd685f96-5fk8w"] Oct 30 18:42:42.240671 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:42.240640 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:42.274188 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:42.272247 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-9aded813\"" Oct 30 18:42:42.355263 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:42.355230 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/272ba07f-785c-4d8f-b2e4-93ca7abf4924-plugins-config-volume\") pod \"payload-processing-e2e-ait-9aded813-86dd685f96-5fk8w\" (UID: \"272ba07f-785c-4d8f-b2e4-93ca7abf4924\") " pod="openshift-ingress/payload-processing-e2e-ait-9aded813-86dd685f96-5fk8w" Oct 30 18:42:42.357370 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:42.355342 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6q8ck\" (UniqueName: \"kubernetes.io/projected/272ba07f-785c-4d8f-b2e4-93ca7abf4924-kube-api-access-6q8ck\") pod \"payload-processing-e2e-ait-9aded813-86dd685f96-5fk8w\" (UID: \"272ba07f-785c-4d8f-b2e4-93ca7abf4924\") " pod="openshift-ingress/payload-processing-e2e-ait-9aded813-86dd685f96-5fk8w" Oct 30 18:42:42.456350 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:42.456276 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/272ba07f-785c-4d8f-b2e4-93ca7abf4924-plugins-config-volume\") pod \"payload-processing-e2e-ait-9aded813-86dd685f96-5fk8w\" (UID: \"272ba07f-785c-4d8f-b2e4-93ca7abf4924\") " pod="openshift-ingress/payload-processing-e2e-ait-9aded813-86dd685f96-5fk8w" Oct 30 18:42:42.456350 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:42.456333 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6q8ck\" (UniqueName: \"kubernetes.io/projected/272ba07f-785c-4d8f-b2e4-93ca7abf4924-kube-api-access-6q8ck\") pod \"payload-processing-e2e-ait-9aded813-86dd685f96-5fk8w\" (UID: \"272ba07f-785c-4d8f-b2e4-93ca7abf4924\") " pod="openshift-ingress/payload-processing-e2e-ait-9aded813-86dd685f96-5fk8w" Oct 30 18:42:42.456978 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:42.456958 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/272ba07f-785c-4d8f-b2e4-93ca7abf4924-plugins-config-volume\") pod \"payload-processing-e2e-ait-9aded813-86dd685f96-5fk8w\" (UID: \"272ba07f-785c-4d8f-b2e4-93ca7abf4924\") " pod="openshift-ingress/payload-processing-e2e-ait-9aded813-86dd685f96-5fk8w" Oct 30 18:42:42.469071 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:42.469022 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6q8ck\" (UniqueName: \"kubernetes.io/projected/272ba07f-785c-4d8f-b2e4-93ca7abf4924-kube-api-access-6q8ck\") pod \"payload-processing-e2e-ait-9aded813-86dd685f96-5fk8w\" (UID: \"272ba07f-785c-4d8f-b2e4-93ca7abf4924\") " pod="openshift-ingress/payload-processing-e2e-ait-9aded813-86dd685f96-5fk8w" Oct 30 18:42:42.514224 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:42.514189 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9aded813-86dd685f96-5fk8w" Oct 30 18:42:42.802463 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:42.802442 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9aded813-86dd685f96-5fk8w"] Oct 30 18:42:42.804601 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:42:42.804571 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod272ba07f_785c_4d8f_b2e4_93ca7abf4924.slice/crio-4e3a5403b960a2e5f6675d6649a75b3f38bc6f2b16923ebe367fc2e3b533e806 WatchSource:0}: Error finding container 4e3a5403b960a2e5f6675d6649a75b3f38bc6f2b16923ebe367fc2e3b533e806: Status 404 returned error can't find the container with id 4e3a5403b960a2e5f6675d6649a75b3f38bc6f2b16923ebe367fc2e3b533e806 Oct 30 18:42:42.929870 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:42.929836 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:43.073052 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:43.073019 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9aded813-86dd685f96-5fk8w" event={"ID":"272ba07f-785c-4d8f-b2e4-93ca7abf4924","Type":"ContainerStarted","Data":"4e3a5403b960a2e5f6675d6649a75b3f38bc6f2b16923ebe367fc2e3b533e806"} Oct 30 18:42:43.754186 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:43.754132 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:43.869527 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:43.869493 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d7289525-84c89c55f8-n8qlc"] Oct 30 18:42:43.873764 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:43.873740 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7289525-84c89c55f8-n8qlc" Oct 30 18:42:43.883547 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:43.882924 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-d7289525-serving-cert\"" Oct 30 18:42:43.897188 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:43.897114 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d7289525-84c89c55f8-n8qlc"] Oct 30 18:42:43.926708 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:43.926626 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:43.972378 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:43.972338 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3ef8346-6816-48c6-9c96-8ba3b9f361ef-openshift-service-ca\") pod \"maas-api-e2e-ait-d7289525-84c89c55f8-n8qlc\" (UID: \"a3ef8346-6816-48c6-9c96-8ba3b9f361ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7289525-84c89c55f8-n8qlc" Oct 30 18:42:43.972567 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:43.972405 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3ef8346-6816-48c6-9c96-8ba3b9f361ef-maas-api-tls\") pod \"maas-api-e2e-ait-d7289525-84c89c55f8-n8qlc\" (UID: \"a3ef8346-6816-48c6-9c96-8ba3b9f361ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7289525-84c89c55f8-n8qlc" Oct 30 18:42:43.972837 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:43.972560 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3ef8346-6816-48c6-9c96-8ba3b9f361ef-metrics-certs\") pod \"maas-api-e2e-ait-d7289525-84c89c55f8-n8qlc\" (UID: \"a3ef8346-6816-48c6-9c96-8ba3b9f361ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7289525-84c89c55f8-n8qlc" Oct 30 18:42:43.972837 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:43.972678 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz28r\" (UniqueName: \"kubernetes.io/projected/a3ef8346-6816-48c6-9c96-8ba3b9f361ef-kube-api-access-pz28r\") pod \"maas-api-e2e-ait-d7289525-84c89c55f8-n8qlc\" (UID: \"a3ef8346-6816-48c6-9c96-8ba3b9f361ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7289525-84c89c55f8-n8qlc" Oct 30 18:42:44.047709 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:44.047615 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:44.074340 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:44.074306 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz28r\" (UniqueName: \"kubernetes.io/projected/a3ef8346-6816-48c6-9c96-8ba3b9f361ef-kube-api-access-pz28r\") pod \"maas-api-e2e-ait-d7289525-84c89c55f8-n8qlc\" (UID: \"a3ef8346-6816-48c6-9c96-8ba3b9f361ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7289525-84c89c55f8-n8qlc" Oct 30 18:42:44.074583 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:44.074373 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3ef8346-6816-48c6-9c96-8ba3b9f361ef-openshift-service-ca\") pod \"maas-api-e2e-ait-d7289525-84c89c55f8-n8qlc\" (UID: \"a3ef8346-6816-48c6-9c96-8ba3b9f361ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7289525-84c89c55f8-n8qlc" Oct 30 18:42:44.074583 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:44.074441 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3ef8346-6816-48c6-9c96-8ba3b9f361ef-maas-api-tls\") pod \"maas-api-e2e-ait-d7289525-84c89c55f8-n8qlc\" (UID: \"a3ef8346-6816-48c6-9c96-8ba3b9f361ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7289525-84c89c55f8-n8qlc" Oct 30 18:42:44.074583 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:44.074510 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3ef8346-6816-48c6-9c96-8ba3b9f361ef-metrics-certs\") pod \"maas-api-e2e-ait-d7289525-84c89c55f8-n8qlc\" (UID: \"a3ef8346-6816-48c6-9c96-8ba3b9f361ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7289525-84c89c55f8-n8qlc" Oct 30 18:42:44.075696 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:44.075672 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3ef8346-6816-48c6-9c96-8ba3b9f361ef-openshift-service-ca\") pod \"maas-api-e2e-ait-d7289525-84c89c55f8-n8qlc\" (UID: \"a3ef8346-6816-48c6-9c96-8ba3b9f361ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7289525-84c89c55f8-n8qlc" Oct 30 18:42:44.077981 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:44.077955 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3ef8346-6816-48c6-9c96-8ba3b9f361ef-metrics-certs\") pod \"maas-api-e2e-ait-d7289525-84c89c55f8-n8qlc\" (UID: \"a3ef8346-6816-48c6-9c96-8ba3b9f361ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7289525-84c89c55f8-n8qlc" Oct 30 18:42:44.077981 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:44.077955 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3ef8346-6816-48c6-9c96-8ba3b9f361ef-maas-api-tls\") pod \"maas-api-e2e-ait-d7289525-84c89c55f8-n8qlc\" (UID: \"a3ef8346-6816-48c6-9c96-8ba3b9f361ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7289525-84c89c55f8-n8qlc" Oct 30 18:42:44.086660 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:44.086633 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz28r\" (UniqueName: \"kubernetes.io/projected/a3ef8346-6816-48c6-9c96-8ba3b9f361ef-kube-api-access-pz28r\") pod \"maas-api-e2e-ait-d7289525-84c89c55f8-n8qlc\" (UID: \"a3ef8346-6816-48c6-9c96-8ba3b9f361ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7289525-84c89c55f8-n8qlc" Oct 30 18:42:44.164079 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:44.164018 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:44.192978 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:44.192947 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7289525-84c89c55f8-n8qlc" Oct 30 18:42:44.476103 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:44.476077 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d7289525-84c89c55f8-n8qlc"] Oct 30 18:42:44.488893 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:42:44.488857 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3ef8346_6816_48c6_9c96_8ba3b9f361ef.slice/crio-e955ba883c3901420573a5c6d2351c58cdffbdad14d9b85e40525c16887d986c WatchSource:0}: Error finding container e955ba883c3901420573a5c6d2351c58cdffbdad14d9b85e40525c16887d986c: Status 404 returned error can't find the container with id e955ba883c3901420573a5c6d2351c58cdffbdad14d9b85e40525c16887d986c Oct 30 18:42:44.626259 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:44.626169 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:44.809373 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:44.809343 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:45.024338 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:45.024293 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:45.089312 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:45.089213 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7289525-84c89c55f8-n8qlc" event={"ID":"a3ef8346-6816-48c6-9c96-8ba3b9f361ef","Type":"ContainerStarted","Data":"2bccc4fae6371de42c0a3f8c2bc5f61c8921cc29df7b25c998c200c1ce44ebe5"} Oct 30 18:42:45.089312 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:45.089287 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7289525-84c89c55f8-n8qlc" event={"ID":"a3ef8346-6816-48c6-9c96-8ba3b9f361ef","Type":"ContainerStarted","Data":"e955ba883c3901420573a5c6d2351c58cdffbdad14d9b85e40525c16887d986c"} Oct 30 18:42:45.089544 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:45.089535 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7289525-84c89c55f8-n8qlc" Oct 30 18:42:45.117312 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:45.117253 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7289525-84c89c55f8-n8qlc" podStartSLOduration=1.6362523740000001 podStartE2EDuration="2.117236686s" totalImagesPullingTime="480.984312ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:42:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:42:45.114333581 +0000 UTC m=+532.643364719" watchObservedRunningTime="2026-10-30 18:42:45.117236686 +0000 UTC m=+532.646267824" Oct 30 18:42:45.146390 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:45.146352 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:45.248360 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:45.248282 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:46.044654 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:46.044611 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:46.097892 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:46.097826 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9aded813-86dd685f96-5fk8w" event={"ID":"272ba07f-785c-4d8f-b2e4-93ca7abf4924","Type":"ContainerStarted","Data":"4b03264e4e94a3ece93153047085a9f1a6b4128917370c65daee0fc791d3bdcf"} Oct 30 18:42:46.098061 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:46.097928 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-9aded813-86dd685f96-5fk8w" Oct 30 18:42:46.117080 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:46.117039 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-9aded813-86dd685f96-5fk8w" podStartSLOduration=1.6317838180000002 podStartE2EDuration="4.117025411s" totalImagesPullingTime="2.485241593s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:42:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:42:46.115347872 +0000 UTC m=+533.644379009" watchObservedRunningTime="2026-10-30 18:42:46.117025411 +0000 UTC m=+533.646056552" Oct 30 18:42:47.181355 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:47.181325 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:47.911794 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:47.911760 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:48.036561 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:48.036503 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:48.079301 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:48.079275 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f5768279-5cfb5444bd-mdhlb" Oct 30 18:42:49.023695 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:49.023642 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:49.165779 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:49.165729 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:49.284548 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:49.284463 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:49.954425 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:49.954393 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:50.067132 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:50.067102 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:50.931318 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:50.931283 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:51.104121 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:51.104096 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7289525-84c89c55f8-n8qlc" Oct 30 18:42:51.856888 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:51.856698 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-72bc98fe-7ddbc86959-p6tch"] Oct 30 18:42:51.861454 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:51.861430 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-72bc98fe-7ddbc86959-p6tch" Oct 30 18:42:51.871936 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:51.871719 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-72bc98fe-serving-cert\"" Oct 30 18:42:51.877760 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:51.877737 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-72bc98fe-7ddbc86959-p6tch"] Oct 30 18:42:51.950215 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:51.950173 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:51.965587 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:51.965559 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spc8v\" (UniqueName: \"kubernetes.io/projected/b64ba209-e869-4bb6-abdd-db517312ea7f-kube-api-access-spc8v\") pod \"maas-api-e2e-derive-72bc98fe-7ddbc86959-p6tch\" (UID: \"b64ba209-e869-4bb6-abdd-db517312ea7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-72bc98fe-7ddbc86959-p6tch" Oct 30 18:42:51.965713 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:51.965597 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b64ba209-e869-4bb6-abdd-db517312ea7f-metrics-certs\") pod \"maas-api-e2e-derive-72bc98fe-7ddbc86959-p6tch\" (UID: \"b64ba209-e869-4bb6-abdd-db517312ea7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-72bc98fe-7ddbc86959-p6tch" Oct 30 18:42:51.965768 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:51.965719 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b64ba209-e869-4bb6-abdd-db517312ea7f-maas-api-tls\") pod \"maas-api-e2e-derive-72bc98fe-7ddbc86959-p6tch\" (UID: \"b64ba209-e869-4bb6-abdd-db517312ea7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-72bc98fe-7ddbc86959-p6tch" Oct 30 18:42:51.965858 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:51.965808 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b64ba209-e869-4bb6-abdd-db517312ea7f-openshift-service-ca\") pod \"maas-api-e2e-derive-72bc98fe-7ddbc86959-p6tch\" (UID: \"b64ba209-e869-4bb6-abdd-db517312ea7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-72bc98fe-7ddbc86959-p6tch" Oct 30 18:42:52.065000 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:52.064966 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:52.067253 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:52.067221 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spc8v\" (UniqueName: \"kubernetes.io/projected/b64ba209-e869-4bb6-abdd-db517312ea7f-kube-api-access-spc8v\") pod \"maas-api-e2e-derive-72bc98fe-7ddbc86959-p6tch\" (UID: \"b64ba209-e869-4bb6-abdd-db517312ea7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-72bc98fe-7ddbc86959-p6tch" Oct 30 18:42:52.067478 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:52.067451 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b64ba209-e869-4bb6-abdd-db517312ea7f-metrics-certs\") pod \"maas-api-e2e-derive-72bc98fe-7ddbc86959-p6tch\" (UID: \"b64ba209-e869-4bb6-abdd-db517312ea7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-72bc98fe-7ddbc86959-p6tch" Oct 30 18:42:52.067610 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:52.067541 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b64ba209-e869-4bb6-abdd-db517312ea7f-maas-api-tls\") pod \"maas-api-e2e-derive-72bc98fe-7ddbc86959-p6tch\" (UID: \"b64ba209-e869-4bb6-abdd-db517312ea7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-72bc98fe-7ddbc86959-p6tch" Oct 30 18:42:52.067610 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:52.067575 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b64ba209-e869-4bb6-abdd-db517312ea7f-openshift-service-ca\") pod \"maas-api-e2e-derive-72bc98fe-7ddbc86959-p6tch\" (UID: \"b64ba209-e869-4bb6-abdd-db517312ea7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-72bc98fe-7ddbc86959-p6tch" Oct 30 18:42:52.068674 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:52.068649 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b64ba209-e869-4bb6-abdd-db517312ea7f-openshift-service-ca\") pod \"maas-api-e2e-derive-72bc98fe-7ddbc86959-p6tch\" (UID: \"b64ba209-e869-4bb6-abdd-db517312ea7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-72bc98fe-7ddbc86959-p6tch" Oct 30 18:42:52.070702 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:52.070676 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b64ba209-e869-4bb6-abdd-db517312ea7f-metrics-certs\") pod \"maas-api-e2e-derive-72bc98fe-7ddbc86959-p6tch\" (UID: \"b64ba209-e869-4bb6-abdd-db517312ea7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-72bc98fe-7ddbc86959-p6tch" Oct 30 18:42:52.070909 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:52.070886 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b64ba209-e869-4bb6-abdd-db517312ea7f-maas-api-tls\") pod \"maas-api-e2e-derive-72bc98fe-7ddbc86959-p6tch\" (UID: \"b64ba209-e869-4bb6-abdd-db517312ea7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-72bc98fe-7ddbc86959-p6tch" Oct 30 18:42:52.079277 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:52.079246 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spc8v\" (UniqueName: \"kubernetes.io/projected/b64ba209-e869-4bb6-abdd-db517312ea7f-kube-api-access-spc8v\") pod \"maas-api-e2e-derive-72bc98fe-7ddbc86959-p6tch\" (UID: \"b64ba209-e869-4bb6-abdd-db517312ea7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-72bc98fe-7ddbc86959-p6tch" Oct 30 18:42:52.179849 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:52.179777 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-72bc98fe-7ddbc86959-p6tch" Oct 30 18:42:52.487816 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:52.487790 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-72bc98fe-7ddbc86959-p6tch"] Oct 30 18:42:52.490527 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:42:52.490499 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb64ba209_e869_4bb6_abdd_db517312ea7f.slice/crio-511632ccbd7ab34b81e86b56bb37b7a38335ee09fa9b66d1b63061502ede4844 WatchSource:0}: Error finding container 511632ccbd7ab34b81e86b56bb37b7a38335ee09fa9b66d1b63061502ede4844: Status 404 returned error can't find the container with id 511632ccbd7ab34b81e86b56bb37b7a38335ee09fa9b66d1b63061502ede4844 Oct 30 18:42:53.048536 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:53.048498 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:53.135999 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:53.135910 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-72bc98fe-7ddbc86959-p6tch" event={"ID":"b64ba209-e869-4bb6-abdd-db517312ea7f","Type":"ContainerStarted","Data":"1550b2b3bbdb3edfbe8179b850ed2046b1acd9b3af627808e2e841ffafb01e9a"} Oct 30 18:42:53.135999 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:53.135968 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-72bc98fe-7ddbc86959-p6tch" event={"ID":"b64ba209-e869-4bb6-abdd-db517312ea7f","Type":"ContainerStarted","Data":"511632ccbd7ab34b81e86b56bb37b7a38335ee09fa9b66d1b63061502ede4844"} Oct 30 18:42:53.136237 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:53.136048 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-72bc98fe-7ddbc86959-p6tch" Oct 30 18:42:53.161778 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:53.161727 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-72bc98fe-7ddbc86959-p6tch" podStartSLOduration=1.7702697170000001 podStartE2EDuration="2.161713024s" totalImagesPullingTime="391.443307ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:42:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:42:53.161394167 +0000 UTC m=+540.690425305" watchObservedRunningTime="2026-10-30 18:42:53.161713024 +0000 UTC m=+540.690744186" Oct 30 18:42:53.932106 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:42:53.932037 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 18:42:54.111389 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:54.111353 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:55.059056 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:55.059025 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:56.024223 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:56.024186 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:57.089457 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:57.089423 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:57.105639 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:57.105616 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-9aded813-86dd685f96-5fk8w" Oct 30 18:42:57.904823 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:57.904782 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:58.803902 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:58.803866 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:59.147244 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:59.147217 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-72bc98fe-7ddbc86959-p6tch" Oct 30 18:42:59.801072 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:42:59.801034 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:00.808376 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:00.808347 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:01.801089 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:01.801050 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:02.770476 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:02.770442 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:03.620318 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:03.620238 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:04.466938 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:04.466906 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:05.308860 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:05.308821 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:06.217116 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:06.217079 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:07.140218 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:07.140122 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:07.929797 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:07.929755 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:08.924290 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:08.924255 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:09.420683 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:09.420650 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:09.526751 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:09.526720 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:09.641835 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:09.641801 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:09.753318 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:09.753243 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:10.008728 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:10.008644 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:10.118200 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:10.118170 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:10.277407 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:10.277314 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:10.802488 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:10.802460 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d7289525-84c89c55f8-n8qlc"] Oct 30 18:43:10.803008 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:10.802682 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7289525-84c89c55f8-n8qlc" podUID="a3ef8346-6816-48c6-9c96-8ba3b9f361ef" containerName="maas-api" containerID="cri-o://2bccc4fae6371de42c0a3f8c2bc5f61c8921cc29df7b25c998c200c1ce44ebe5" gracePeriod=30 Oct 30 18:43:10.864642 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:10.864605 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:11.055257 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:11.055220 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:11.089694 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:11.089669 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9aded813-86dd685f96-5fk8w"] Oct 30 18:43:11.116484 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:11.116456 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7289525-84c89c55f8-n8qlc" Oct 30 18:43:11.181897 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:11.181849 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:11.231190 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:11.231163 6960 generic.go:356] "Generic (PLEG): container finished" podID="a3ef8346-6816-48c6-9c96-8ba3b9f361ef" containerID="2bccc4fae6371de42c0a3f8c2bc5f61c8921cc29df7b25c998c200c1ce44ebe5" exitCode=0 Oct 30 18:43:11.231352 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:11.231211 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7289525-84c89c55f8-n8qlc" Oct 30 18:43:11.231352 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:11.231248 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7289525-84c89c55f8-n8qlc" event={"ID":"a3ef8346-6816-48c6-9c96-8ba3b9f361ef","Type":"ContainerDied","Data":"2bccc4fae6371de42c0a3f8c2bc5f61c8921cc29df7b25c998c200c1ce44ebe5"} Oct 30 18:43:11.231352 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:11.231307 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7289525-84c89c55f8-n8qlc" event={"ID":"a3ef8346-6816-48c6-9c96-8ba3b9f361ef","Type":"ContainerDied","Data":"e955ba883c3901420573a5c6d2351c58cdffbdad14d9b85e40525c16887d986c"} Oct 30 18:43:11.231352 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:11.231336 6960 scope.go:122] "RemoveContainer" containerID="2bccc4fae6371de42c0a3f8c2bc5f61c8921cc29df7b25c998c200c1ce44ebe5" Oct 30 18:43:11.240298 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:11.240279 6960 scope.go:122] "RemoveContainer" containerID="2bccc4fae6371de42c0a3f8c2bc5f61c8921cc29df7b25c998c200c1ce44ebe5" Oct 30 18:43:11.240548 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:43:11.240528 6960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bccc4fae6371de42c0a3f8c2bc5f61c8921cc29df7b25c998c200c1ce44ebe5\": container with ID starting with 2bccc4fae6371de42c0a3f8c2bc5f61c8921cc29df7b25c998c200c1ce44ebe5 not found: ID does not exist" containerID="2bccc4fae6371de42c0a3f8c2bc5f61c8921cc29df7b25c998c200c1ce44ebe5" Oct 30 18:43:11.240604 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:11.240558 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bccc4fae6371de42c0a3f8c2bc5f61c8921cc29df7b25c998c200c1ce44ebe5"} err="failed to get container status \"2bccc4fae6371de42c0a3f8c2bc5f61c8921cc29df7b25c998c200c1ce44ebe5\": rpc error: code = NotFound desc = could not find container \"2bccc4fae6371de42c0a3f8c2bc5f61c8921cc29df7b25c998c200c1ce44ebe5\": container with ID starting with 2bccc4fae6371de42c0a3f8c2bc5f61c8921cc29df7b25c998c200c1ce44ebe5 not found: ID does not exist" Oct 30 18:43:11.260970 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:11.260945 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3ef8346-6816-48c6-9c96-8ba3b9f361ef-kube-api-access-pz28r\" (UniqueName: \"kubernetes.io/projected/a3ef8346-6816-48c6-9c96-8ba3b9f361ef-kube-api-access-pz28r\") pod \"a3ef8346-6816-48c6-9c96-8ba3b9f361ef\" (UID: \"a3ef8346-6816-48c6-9c96-8ba3b9f361ef\") " Oct 30 18:43:11.261061 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:11.260995 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3ef8346-6816-48c6-9c96-8ba3b9f361ef-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3ef8346-6816-48c6-9c96-8ba3b9f361ef-metrics-certs\") pod \"a3ef8346-6816-48c6-9c96-8ba3b9f361ef\" (UID: \"a3ef8346-6816-48c6-9c96-8ba3b9f361ef\") " Oct 30 18:43:11.261110 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:11.261061 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3ef8346-6816-48c6-9c96-8ba3b9f361ef-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3ef8346-6816-48c6-9c96-8ba3b9f361ef-openshift-service-ca\") pod \"a3ef8346-6816-48c6-9c96-8ba3b9f361ef\" (UID: \"a3ef8346-6816-48c6-9c96-8ba3b9f361ef\") " Oct 30 18:43:11.261110 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:11.261079 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3ef8346-6816-48c6-9c96-8ba3b9f361ef-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3ef8346-6816-48c6-9c96-8ba3b9f361ef-maas-api-tls\") pod \"a3ef8346-6816-48c6-9c96-8ba3b9f361ef\" (UID: \"a3ef8346-6816-48c6-9c96-8ba3b9f361ef\") " Oct 30 18:43:11.261439 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:11.261410 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3ef8346-6816-48c6-9c96-8ba3b9f361ef-openshift-service-ca" pod "a3ef8346-6816-48c6-9c96-8ba3b9f361ef" (UID: "a3ef8346-6816-48c6-9c96-8ba3b9f361ef"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:43:11.261562 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:11.261541 6960 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3ef8346-6816-48c6-9c96-8ba3b9f361ef-openshift-service-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:43:11.263208 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:11.263129 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3ef8346-6816-48c6-9c96-8ba3b9f361ef-kube-api-access-pz28r" pod "a3ef8346-6816-48c6-9c96-8ba3b9f361ef" (UID: "a3ef8346-6816-48c6-9c96-8ba3b9f361ef"). InnerVolumeSpecName "kube-api-access-pz28r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:43:11.263284 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:11.263259 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3ef8346-6816-48c6-9c96-8ba3b9f361ef-maas-api-tls" pod "a3ef8346-6816-48c6-9c96-8ba3b9f361ef" (UID: "a3ef8346-6816-48c6-9c96-8ba3b9f361ef"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:43:11.263284 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:11.263269 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3ef8346-6816-48c6-9c96-8ba3b9f361ef-metrics-certs" pod "a3ef8346-6816-48c6-9c96-8ba3b9f361ef" (UID: "a3ef8346-6816-48c6-9c96-8ba3b9f361ef"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:43:11.292575 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:11.292546 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:11.362671 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:11.362588 6960 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3ef8346-6816-48c6-9c96-8ba3b9f361ef-maas-api-tls\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:43:11.362671 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:11.362620 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pz28r\" (UniqueName: \"kubernetes.io/projected/a3ef8346-6816-48c6-9c96-8ba3b9f361ef-kube-api-access-pz28r\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:43:11.362671 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:11.362653 6960 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3ef8346-6816-48c6-9c96-8ba3b9f361ef-metrics-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:43:11.560166 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:11.558324 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d7289525-84c89c55f8-n8qlc"] Oct 30 18:43:11.562864 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:11.562801 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d7289525-84c89c55f8-n8qlc"] Oct 30 18:43:11.632219 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:11.632104 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:11.748783 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:11.748747 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:11.881162 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:11.881111 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:12.592721 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:12.592687 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3ef8346-6816-48c6-9c96-8ba3b9f361ef" path="/var/lib/kubelet/pods/a3ef8346-6816-48c6-9c96-8ba3b9f361ef/volumes" Oct 30 18:43:12.624315 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:12.624285 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:13.029175 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:13.029074 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:13.622427 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:13.622394 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:13.871502 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:13.871465 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:13.983702 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:13.983614 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:14.098341 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:14.098299 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d7289525-84c89c55f8-n8qlc" podUID="a3ef8346-6816-48c6-9c96-8ba3b9f361ef" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.41:8443/health\": context deadline exceeded" Oct 30 18:43:14.383322 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:14.383260 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:14.498086 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:14.498048 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:14.548882 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:14.548846 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f5768279-5cfb5444bd-mdhlb"] Oct 30 18:43:14.549258 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:14.549225 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f5768279-5cfb5444bd-mdhlb" podUID="ac740be5-4f91-4988-8321-b73ea6d7a842" containerName="maas-api" containerID="cri-o://71e81e6a74177e98a180b2b44760f0aa7e614f36f9a11cedf7fc9096182e62ec" gracePeriod=30 Oct 30 18:43:14.624965 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:14.624923 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:14.763969 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:14.763933 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:14.833700 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:14.833680 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f5768279-5cfb5444bd-mdhlb" Oct 30 18:43:14.906558 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:14.906531 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac740be5-4f91-4988-8321-b73ea6d7a842-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac740be5-4f91-4988-8321-b73ea6d7a842-openshift-service-ca\") pod \"ac740be5-4f91-4988-8321-b73ea6d7a842\" (UID: \"ac740be5-4f91-4988-8321-b73ea6d7a842\") " Oct 30 18:43:14.906731 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:14.906572 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac740be5-4f91-4988-8321-b73ea6d7a842-kube-api-access-g5hxm\" (UniqueName: \"kubernetes.io/projected/ac740be5-4f91-4988-8321-b73ea6d7a842-kube-api-access-g5hxm\") pod \"ac740be5-4f91-4988-8321-b73ea6d7a842\" (UID: \"ac740be5-4f91-4988-8321-b73ea6d7a842\") " Oct 30 18:43:14.906731 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:14.906614 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac740be5-4f91-4988-8321-b73ea6d7a842-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac740be5-4f91-4988-8321-b73ea6d7a842-metrics-certs\") pod \"ac740be5-4f91-4988-8321-b73ea6d7a842\" (UID: \"ac740be5-4f91-4988-8321-b73ea6d7a842\") " Oct 30 18:43:14.906731 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:14.906637 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac740be5-4f91-4988-8321-b73ea6d7a842-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac740be5-4f91-4988-8321-b73ea6d7a842-maas-api-tls\") pod \"ac740be5-4f91-4988-8321-b73ea6d7a842\" (UID: \"ac740be5-4f91-4988-8321-b73ea6d7a842\") " Oct 30 18:43:14.906940 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:14.906908 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac740be5-4f91-4988-8321-b73ea6d7a842-openshift-service-ca" pod "ac740be5-4f91-4988-8321-b73ea6d7a842" (UID: "ac740be5-4f91-4988-8321-b73ea6d7a842"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:43:14.907052 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:14.907031 6960 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac740be5-4f91-4988-8321-b73ea6d7a842-openshift-service-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:43:14.908822 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:14.908792 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac740be5-4f91-4988-8321-b73ea6d7a842-metrics-certs" pod "ac740be5-4f91-4988-8321-b73ea6d7a842" (UID: "ac740be5-4f91-4988-8321-b73ea6d7a842"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:43:14.908910 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:14.908880 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac740be5-4f91-4988-8321-b73ea6d7a842-maas-api-tls" pod "ac740be5-4f91-4988-8321-b73ea6d7a842" (UID: "ac740be5-4f91-4988-8321-b73ea6d7a842"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:43:14.908970 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:14.908954 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac740be5-4f91-4988-8321-b73ea6d7a842-kube-api-access-g5hxm" pod "ac740be5-4f91-4988-8321-b73ea6d7a842" (UID: "ac740be5-4f91-4988-8321-b73ea6d7a842"). InnerVolumeSpecName "kube-api-access-g5hxm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:43:14.933578 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:14.933525 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:15.007802 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:15.007774 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5hxm\" (UniqueName: \"kubernetes.io/projected/ac740be5-4f91-4988-8321-b73ea6d7a842-kube-api-access-g5hxm\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:43:15.007930 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:15.007807 6960 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac740be5-4f91-4988-8321-b73ea6d7a842-metrics-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:43:15.007930 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:15.007823 6960 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac740be5-4f91-4988-8321-b73ea6d7a842-maas-api-tls\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:43:15.049217 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:15.049193 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:15.182612 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:15.182572 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:15.253818 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:15.253746 6960 generic.go:356] "Generic (PLEG): container finished" podID="ac740be5-4f91-4988-8321-b73ea6d7a842" containerID="71e81e6a74177e98a180b2b44760f0aa7e614f36f9a11cedf7fc9096182e62ec" exitCode=0 Oct 30 18:43:15.253955 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:15.253841 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f5768279-5cfb5444bd-mdhlb" Oct 30 18:43:15.253955 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:15.253840 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f5768279-5cfb5444bd-mdhlb" event={"ID":"ac740be5-4f91-4988-8321-b73ea6d7a842","Type":"ContainerDied","Data":"71e81e6a74177e98a180b2b44760f0aa7e614f36f9a11cedf7fc9096182e62ec"} Oct 30 18:43:15.253955 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:15.253896 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f5768279-5cfb5444bd-mdhlb" event={"ID":"ac740be5-4f91-4988-8321-b73ea6d7a842","Type":"ContainerDied","Data":"7bd811383032b0332e343918d7ea98f878cc07366cf0ad29c343ea873358de12"} Oct 30 18:43:15.253955 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:15.253919 6960 scope.go:122] "RemoveContainer" containerID="71e81e6a74177e98a180b2b44760f0aa7e614f36f9a11cedf7fc9096182e62ec" Oct 30 18:43:15.264001 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:15.263987 6960 scope.go:122] "RemoveContainer" containerID="71e81e6a74177e98a180b2b44760f0aa7e614f36f9a11cedf7fc9096182e62ec" Oct 30 18:43:15.264274 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:43:15.264258 6960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71e81e6a74177e98a180b2b44760f0aa7e614f36f9a11cedf7fc9096182e62ec\": container with ID starting with 71e81e6a74177e98a180b2b44760f0aa7e614f36f9a11cedf7fc9096182e62ec not found: ID does not exist" containerID="71e81e6a74177e98a180b2b44760f0aa7e614f36f9a11cedf7fc9096182e62ec" Oct 30 18:43:15.264343 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:15.264287 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71e81e6a74177e98a180b2b44760f0aa7e614f36f9a11cedf7fc9096182e62ec"} err="failed to get container status \"71e81e6a74177e98a180b2b44760f0aa7e614f36f9a11cedf7fc9096182e62ec\": rpc error: code = NotFound desc = could not find container \"71e81e6a74177e98a180b2b44760f0aa7e614f36f9a11cedf7fc9096182e62ec\": container with ID starting with 71e81e6a74177e98a180b2b44760f0aa7e614f36f9a11cedf7fc9096182e62ec not found: ID does not exist" Oct 30 18:43:15.278791 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:15.278735 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f5768279-5cfb5444bd-mdhlb"] Oct 30 18:43:15.280784 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:15.280766 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f5768279-5cfb5444bd-mdhlb"] Oct 30 18:43:15.296264 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:15.296237 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:15.560803 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:15.560770 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:16.090452 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:16.090418 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-9aded813-86dd685f96-5fk8w" podUID="272ba07f-785c-4d8f-b2e4-93ca7abf4924" containerName="payload-processing" containerID="cri-o://4b03264e4e94a3ece93153047085a9f1a6b4128917370c65daee0fc791d3bdcf" gracePeriod=25 Oct 30 18:43:16.118184 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:16.118156 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:16.227598 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:16.227560 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:16.265662 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:16.265401 6960 generic.go:356] "Generic (PLEG): container finished" podID="272ba07f-785c-4d8f-b2e4-93ca7abf4924" containerID="4b03264e4e94a3ece93153047085a9f1a6b4128917370c65daee0fc791d3bdcf" exitCode=0 Oct 30 18:43:16.265662 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:16.265613 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9aded813-86dd685f96-5fk8w" event={"ID":"272ba07f-785c-4d8f-b2e4-93ca7abf4924","Type":"ContainerDied","Data":"4b03264e4e94a3ece93153047085a9f1a6b4128917370c65daee0fc791d3bdcf"} Oct 30 18:43:16.339160 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:16.339121 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9aded813-86dd685f96-5fk8w" Oct 30 18:43:16.343722 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:16.343669 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:16.382401 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:16.382373 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-7a8fa2-778fb96bb-tlvqh"] Oct 30 18:43:16.382935 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:16.382917 6960 cpu_manager.go:414] "containerMap: removing container" podUID="a3ef8346-6816-48c6-9c96-8ba3b9f361ef" containerName="maas-api" Oct 30 18:43:16.383007 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:16.382936 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3ef8346-6816-48c6-9c96-8ba3b9f361ef" containerName="maas-api" Oct 30 18:43:16.383007 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:16.382957 6960 cpu_manager.go:414] "containerMap: removing container" podUID="ac740be5-4f91-4988-8321-b73ea6d7a842" containerName="maas-api" Oct 30 18:43:16.383007 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:16.382962 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac740be5-4f91-4988-8321-b73ea6d7a842" containerName="maas-api" Oct 30 18:43:16.383007 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:16.382972 6960 cpu_manager.go:414] "containerMap: removing container" podUID="272ba07f-785c-4d8f-b2e4-93ca7abf4924" containerName="payload-processing" Oct 30 18:43:16.383007 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:16.382977 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="272ba07f-785c-4d8f-b2e4-93ca7abf4924" containerName="payload-processing" Oct 30 18:43:16.383190 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:16.383069 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3ef8346-6816-48c6-9c96-8ba3b9f361ef" containerName="maas-api" Oct 30 18:43:16.383190 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:16.383079 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac740be5-4f91-4988-8321-b73ea6d7a842" containerName="maas-api" Oct 30 18:43:16.383190 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:16.383089 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="272ba07f-785c-4d8f-b2e4-93ca7abf4924" containerName="payload-processing" Oct 30 18:43:16.386283 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:16.386268 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-7a8fa2-778fb96bb-tlvqh" Oct 30 18:43:16.397262 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:16.397241 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-7a8fa2\"" Oct 30 18:43:16.397392 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:16.397287 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-7a8fa2-dockercfg-bjbpb\"" Oct 30 18:43:16.397959 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:16.397936 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-7a8fa2-778fb96bb-tlvqh"] Oct 30 18:43:16.460940 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:16.460904 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:16.519960 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:16.519927 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/272ba07f-785c-4d8f-b2e4-93ca7abf4924-kube-api-access-6q8ck\" (UniqueName: \"kubernetes.io/projected/272ba07f-785c-4d8f-b2e4-93ca7abf4924-kube-api-access-6q8ck\") pod \"272ba07f-785c-4d8f-b2e4-93ca7abf4924\" (UID: \"272ba07f-785c-4d8f-b2e4-93ca7abf4924\") " Oct 30 18:43:16.520186 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:16.520072 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/272ba07f-785c-4d8f-b2e4-93ca7abf4924-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/272ba07f-785c-4d8f-b2e4-93ca7abf4924-plugins-config-volume\") pod \"272ba07f-785c-4d8f-b2e4-93ca7abf4924\" (UID: \"272ba07f-785c-4d8f-b2e4-93ca7abf4924\") " Oct 30 18:43:16.520503 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:16.520465 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67a3f38a-10e8-4826-925c-520ce8297f11-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-7a8fa2-778fb96bb-tlvqh\" (UID: \"67a3f38a-10e8-4826-925c-520ce8297f11\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-7a8fa2-778fb96bb-tlvqh" Oct 30 18:43:16.520603 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:16.520568 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7x7k5\" (UniqueName: \"kubernetes.io/projected/67a3f38a-10e8-4826-925c-520ce8297f11-kube-api-access-7x7k5\") pod \"payload-processing-e2e-worker-w3-7a8fa2-778fb96bb-tlvqh\" (UID: \"67a3f38a-10e8-4826-925c-520ce8297f11\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-7a8fa2-778fb96bb-tlvqh" Oct 30 18:43:16.521602 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:16.521564 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/272ba07f-785c-4d8f-b2e4-93ca7abf4924-plugins-config-volume" pod "272ba07f-785c-4d8f-b2e4-93ca7abf4924" (UID: "272ba07f-785c-4d8f-b2e4-93ca7abf4924"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:43:16.524785 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:16.524753 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/272ba07f-785c-4d8f-b2e4-93ca7abf4924-kube-api-access-6q8ck" pod "272ba07f-785c-4d8f-b2e4-93ca7abf4924" (UID: "272ba07f-785c-4d8f-b2e4-93ca7abf4924"). InnerVolumeSpecName "kube-api-access-6q8ck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:43:16.591840 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:16.591810 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac740be5-4f91-4988-8321-b73ea6d7a842" path="/var/lib/kubelet/pods/ac740be5-4f91-4988-8321-b73ea6d7a842/volumes" Oct 30 18:43:16.622137 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:16.622105 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7x7k5\" (UniqueName: \"kubernetes.io/projected/67a3f38a-10e8-4826-925c-520ce8297f11-kube-api-access-7x7k5\") pod \"payload-processing-e2e-worker-w3-7a8fa2-778fb96bb-tlvqh\" (UID: \"67a3f38a-10e8-4826-925c-520ce8297f11\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-7a8fa2-778fb96bb-tlvqh" Oct 30 18:43:16.622925 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:16.622341 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67a3f38a-10e8-4826-925c-520ce8297f11-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-7a8fa2-778fb96bb-tlvqh\" (UID: \"67a3f38a-10e8-4826-925c-520ce8297f11\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-7a8fa2-778fb96bb-tlvqh" Oct 30 18:43:16.622925 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:16.622444 6960 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/272ba07f-785c-4d8f-b2e4-93ca7abf4924-plugins-config-volume\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:43:16.622925 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:16.622463 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6q8ck\" (UniqueName: \"kubernetes.io/projected/272ba07f-785c-4d8f-b2e4-93ca7abf4924-kube-api-access-6q8ck\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:43:16.623431 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:16.623401 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67a3f38a-10e8-4826-925c-520ce8297f11-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-7a8fa2-778fb96bb-tlvqh\" (UID: \"67a3f38a-10e8-4826-925c-520ce8297f11\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-7a8fa2-778fb96bb-tlvqh" Oct 30 18:43:16.639968 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:16.639940 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x7k5\" (UniqueName: \"kubernetes.io/projected/67a3f38a-10e8-4826-925c-520ce8297f11-kube-api-access-7x7k5\") pod \"payload-processing-e2e-worker-w3-7a8fa2-778fb96bb-tlvqh\" (UID: \"67a3f38a-10e8-4826-925c-520ce8297f11\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-7a8fa2-778fb96bb-tlvqh" Oct 30 18:43:16.689762 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:16.689721 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:16.699561 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:16.699536 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-7a8fa2-778fb96bb-tlvqh" Oct 30 18:43:16.876038 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:16.875947 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:17.005170 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:17.004621 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-7a8fa2-778fb96bb-tlvqh"] Oct 30 18:43:17.027818 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:17.027783 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:17.032090 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:43:17.032061 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67a3f38a_10e8_4826_925c_520ce8297f11.slice/crio-37ca11a25b2fd43b339f3f7101d52aeac9be645b18d60129f36f0eb93b6b7d0a WatchSource:0}: Error finding container 37ca11a25b2fd43b339f3f7101d52aeac9be645b18d60129f36f0eb93b6b7d0a: Status 404 returned error can't find the container with id 37ca11a25b2fd43b339f3f7101d52aeac9be645b18d60129f36f0eb93b6b7d0a Oct 30 18:43:17.184238 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:17.184065 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:17.280173 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:17.278403 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9aded813-86dd685f96-5fk8w" Oct 30 18:43:17.280173 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:17.279098 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9aded813-86dd685f96-5fk8w" event={"ID":"272ba07f-785c-4d8f-b2e4-93ca7abf4924","Type":"ContainerDied","Data":"4e3a5403b960a2e5f6675d6649a75b3f38bc6f2b16923ebe367fc2e3b533e806"} Oct 30 18:43:17.280173 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:17.279184 6960 scope.go:122] "RemoveContainer" containerID="4b03264e4e94a3ece93153047085a9f1a6b4128917370c65daee0fc791d3bdcf" Oct 30 18:43:17.282389 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:17.282365 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-7a8fa2-59479c7948-wtw6n"] Oct 30 18:43:17.287832 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:17.287808 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-7a8fa2-778fb96bb-tlvqh" Oct 30 18:43:17.287958 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:17.287847 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-7a8fa2-778fb96bb-tlvqh" event={"ID":"67a3f38a-10e8-4826-925c-520ce8297f11","Type":"ContainerStarted","Data":"e5d7327234db62163358bdb4e01d1231e9e3d2ea60891475b591eed63452145f"} Oct 30 18:43:17.287958 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:17.287885 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-7a8fa2-778fb96bb-tlvqh" event={"ID":"67a3f38a-10e8-4826-925c-520ce8297f11","Type":"ContainerStarted","Data":"37ca11a25b2fd43b339f3f7101d52aeac9be645b18d60129f36f0eb93b6b7d0a"} Oct 30 18:43:17.288060 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:17.288000 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7a8fa2-59479c7948-wtw6n" Oct 30 18:43:17.293582 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:17.293560 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-7a8fa2-serving-cert\"" Oct 30 18:43:17.307636 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:17.307607 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-7a8fa2-59479c7948-wtw6n"] Oct 30 18:43:17.328541 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:17.328501 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vrgn\" (UniqueName: \"kubernetes.io/projected/da7fcb8d-db2d-4b43-a04f-22f4a9c0757b-kube-api-access-9vrgn\") pod \"maas-api-e2e-worker-w3-7a8fa2-59479c7948-wtw6n\" (UID: \"da7fcb8d-db2d-4b43-a04f-22f4a9c0757b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7a8fa2-59479c7948-wtw6n" Oct 30 18:43:17.328720 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:17.328546 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da7fcb8d-db2d-4b43-a04f-22f4a9c0757b-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-7a8fa2-59479c7948-wtw6n\" (UID: \"da7fcb8d-db2d-4b43-a04f-22f4a9c0757b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7a8fa2-59479c7948-wtw6n" Oct 30 18:43:17.328797 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:17.328764 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da7fcb8d-db2d-4b43-a04f-22f4a9c0757b-metrics-certs\") pod \"maas-api-e2e-worker-w3-7a8fa2-59479c7948-wtw6n\" (UID: \"da7fcb8d-db2d-4b43-a04f-22f4a9c0757b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7a8fa2-59479c7948-wtw6n" Oct 30 18:43:17.328854 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:17.328808 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da7fcb8d-db2d-4b43-a04f-22f4a9c0757b-maas-api-tls\") pod \"maas-api-e2e-worker-w3-7a8fa2-59479c7948-wtw6n\" (UID: \"da7fcb8d-db2d-4b43-a04f-22f4a9c0757b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7a8fa2-59479c7948-wtw6n" Oct 30 18:43:17.342091 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:17.342061 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9aded813-86dd685f96-5fk8w"] Oct 30 18:43:17.345813 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:17.345789 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9aded813-86dd685f96-5fk8w"] Oct 30 18:43:17.369284 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:17.369230 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-7a8fa2-778fb96bb-tlvqh" podStartSLOduration=1.369216651 podStartE2EDuration="1.369216651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:43:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:43:17.367503491 +0000 UTC m=+564.896534634" watchObservedRunningTime="2026-10-30 18:43:17.369216651 +0000 UTC m=+564.898247790" Oct 30 18:43:17.375903 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:17.375869 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:17.429832 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:17.429798 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vrgn\" (UniqueName: \"kubernetes.io/projected/da7fcb8d-db2d-4b43-a04f-22f4a9c0757b-kube-api-access-9vrgn\") pod \"maas-api-e2e-worker-w3-7a8fa2-59479c7948-wtw6n\" (UID: \"da7fcb8d-db2d-4b43-a04f-22f4a9c0757b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7a8fa2-59479c7948-wtw6n" Oct 30 18:43:17.430019 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:17.429842 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da7fcb8d-db2d-4b43-a04f-22f4a9c0757b-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-7a8fa2-59479c7948-wtw6n\" (UID: \"da7fcb8d-db2d-4b43-a04f-22f4a9c0757b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7a8fa2-59479c7948-wtw6n" Oct 30 18:43:17.430019 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:17.429921 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da7fcb8d-db2d-4b43-a04f-22f4a9c0757b-metrics-certs\") pod \"maas-api-e2e-worker-w3-7a8fa2-59479c7948-wtw6n\" (UID: \"da7fcb8d-db2d-4b43-a04f-22f4a9c0757b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7a8fa2-59479c7948-wtw6n" Oct 30 18:43:17.430019 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:17.429966 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da7fcb8d-db2d-4b43-a04f-22f4a9c0757b-maas-api-tls\") pod \"maas-api-e2e-worker-w3-7a8fa2-59479c7948-wtw6n\" (UID: \"da7fcb8d-db2d-4b43-a04f-22f4a9c0757b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7a8fa2-59479c7948-wtw6n" Oct 30 18:43:17.430612 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:17.430586 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da7fcb8d-db2d-4b43-a04f-22f4a9c0757b-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-7a8fa2-59479c7948-wtw6n\" (UID: \"da7fcb8d-db2d-4b43-a04f-22f4a9c0757b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7a8fa2-59479c7948-wtw6n" Oct 30 18:43:17.432550 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:17.432530 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da7fcb8d-db2d-4b43-a04f-22f4a9c0757b-maas-api-tls\") pod \"maas-api-e2e-worker-w3-7a8fa2-59479c7948-wtw6n\" (UID: \"da7fcb8d-db2d-4b43-a04f-22f4a9c0757b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7a8fa2-59479c7948-wtw6n" Oct 30 18:43:17.432626 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:17.432584 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da7fcb8d-db2d-4b43-a04f-22f4a9c0757b-metrics-certs\") pod \"maas-api-e2e-worker-w3-7a8fa2-59479c7948-wtw6n\" (UID: \"da7fcb8d-db2d-4b43-a04f-22f4a9c0757b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7a8fa2-59479c7948-wtw6n" Oct 30 18:43:17.447633 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:17.447580 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vrgn\" (UniqueName: \"kubernetes.io/projected/da7fcb8d-db2d-4b43-a04f-22f4a9c0757b-kube-api-access-9vrgn\") pod \"maas-api-e2e-worker-w3-7a8fa2-59479c7948-wtw6n\" (UID: \"da7fcb8d-db2d-4b43-a04f-22f4a9c0757b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7a8fa2-59479c7948-wtw6n" Oct 30 18:43:17.504833 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:17.504801 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:17.605787 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:17.605341 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7a8fa2-59479c7948-wtw6n" Oct 30 18:43:17.895221 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:17.895135 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-7a8fa2-59479c7948-wtw6n"] Oct 30 18:43:17.900420 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:43:17.900391 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda7fcb8d_db2d_4b43_a04f_22f4a9c0757b.slice/crio-49e8c29c25fde0b0e69e5c701421ed553dde3ea1e6a7c3b1329ca854fabdbc4a WatchSource:0}: Error finding container 49e8c29c25fde0b0e69e5c701421ed553dde3ea1e6a7c3b1329ca854fabdbc4a: Status 404 returned error can't find the container with id 49e8c29c25fde0b0e69e5c701421ed553dde3ea1e6a7c3b1329ca854fabdbc4a Oct 30 18:43:18.092183 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:18.091699 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:18.228538 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:18.228453 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:18.246617 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:18.246576 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-72bc98fe-7ddbc86959-p6tch"] Oct 30 18:43:18.248232 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:18.248201 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-72bc98fe-7ddbc86959-p6tch" podUID="b64ba209-e869-4bb6-abdd-db517312ea7f" containerName="maas-api" containerID="cri-o://1550b2b3bbdb3edfbe8179b850ed2046b1acd9b3af627808e2e841ffafb01e9a" gracePeriod=30 Oct 30 18:43:18.293860 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:18.293820 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7a8fa2-59479c7948-wtw6n" event={"ID":"da7fcb8d-db2d-4b43-a04f-22f4a9c0757b","Type":"ContainerStarted","Data":"49e8c29c25fde0b0e69e5c701421ed553dde3ea1e6a7c3b1329ca854fabdbc4a"} Oct 30 18:43:18.499557 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:18.499524 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:18.502777 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:18.502749 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-72bc98fe-7ddbc86959-p6tch" Oct 30 18:43:18.543042 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:18.543014 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b64ba209-e869-4bb6-abdd-db517312ea7f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b64ba209-e869-4bb6-abdd-db517312ea7f-openshift-service-ca\") pod \"b64ba209-e869-4bb6-abdd-db517312ea7f\" (UID: \"b64ba209-e869-4bb6-abdd-db517312ea7f\") " Oct 30 18:43:18.543272 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:18.543056 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b64ba209-e869-4bb6-abdd-db517312ea7f-kube-api-access-spc8v\" (UniqueName: \"kubernetes.io/projected/b64ba209-e869-4bb6-abdd-db517312ea7f-kube-api-access-spc8v\") pod \"b64ba209-e869-4bb6-abdd-db517312ea7f\" (UID: \"b64ba209-e869-4bb6-abdd-db517312ea7f\") " Oct 30 18:43:18.543272 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:18.543089 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b64ba209-e869-4bb6-abdd-db517312ea7f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b64ba209-e869-4bb6-abdd-db517312ea7f-maas-api-tls\") pod \"b64ba209-e869-4bb6-abdd-db517312ea7f\" (UID: \"b64ba209-e869-4bb6-abdd-db517312ea7f\") " Oct 30 18:43:18.543272 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:18.543261 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b64ba209-e869-4bb6-abdd-db517312ea7f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b64ba209-e869-4bb6-abdd-db517312ea7f-metrics-certs\") pod \"b64ba209-e869-4bb6-abdd-db517312ea7f\" (UID: \"b64ba209-e869-4bb6-abdd-db517312ea7f\") " Oct 30 18:43:18.543455 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:18.543384 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b64ba209-e869-4bb6-abdd-db517312ea7f-openshift-service-ca" pod "b64ba209-e869-4bb6-abdd-db517312ea7f" (UID: "b64ba209-e869-4bb6-abdd-db517312ea7f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:43:18.543782 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:18.543711 6960 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b64ba209-e869-4bb6-abdd-db517312ea7f-openshift-service-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:43:18.545357 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:18.545333 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b64ba209-e869-4bb6-abdd-db517312ea7f-maas-api-tls" pod "b64ba209-e869-4bb6-abdd-db517312ea7f" (UID: "b64ba209-e869-4bb6-abdd-db517312ea7f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:43:18.545461 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:18.545357 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b64ba209-e869-4bb6-abdd-db517312ea7f-kube-api-access-spc8v" pod "b64ba209-e869-4bb6-abdd-db517312ea7f" (UID: "b64ba209-e869-4bb6-abdd-db517312ea7f"). InnerVolumeSpecName "kube-api-access-spc8v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:43:18.545575 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:18.545554 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b64ba209-e869-4bb6-abdd-db517312ea7f-metrics-certs" pod "b64ba209-e869-4bb6-abdd-db517312ea7f" (UID: "b64ba209-e869-4bb6-abdd-db517312ea7f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:43:18.592733 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:18.592704 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="272ba07f-785c-4d8f-b2e4-93ca7abf4924" path="/var/lib/kubelet/pods/272ba07f-785c-4d8f-b2e4-93ca7abf4924/volumes" Oct 30 18:43:18.644716 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:18.644686 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spc8v\" (UniqueName: \"kubernetes.io/projected/b64ba209-e869-4bb6-abdd-db517312ea7f-kube-api-access-spc8v\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:43:18.644716 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:18.644718 6960 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b64ba209-e869-4bb6-abdd-db517312ea7f-maas-api-tls\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:43:18.644967 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:18.644732 6960 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b64ba209-e869-4bb6-abdd-db517312ea7f-metrics-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:43:18.877297 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:18.877262 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:19.302972 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:19.302873 6960 generic.go:356] "Generic (PLEG): container finished" podID="b64ba209-e869-4bb6-abdd-db517312ea7f" containerID="1550b2b3bbdb3edfbe8179b850ed2046b1acd9b3af627808e2e841ffafb01e9a" exitCode=0 Oct 30 18:43:19.302972 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:19.302928 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-72bc98fe-7ddbc86959-p6tch" Oct 30 18:43:19.302972 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:19.302930 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-72bc98fe-7ddbc86959-p6tch" event={"ID":"b64ba209-e869-4bb6-abdd-db517312ea7f","Type":"ContainerDied","Data":"1550b2b3bbdb3edfbe8179b850ed2046b1acd9b3af627808e2e841ffafb01e9a"} Oct 30 18:43:19.303546 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:19.302993 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-72bc98fe-7ddbc86959-p6tch" event={"ID":"b64ba209-e869-4bb6-abdd-db517312ea7f","Type":"ContainerDied","Data":"511632ccbd7ab34b81e86b56bb37b7a38335ee09fa9b66d1b63061502ede4844"} Oct 30 18:43:19.303546 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:19.303026 6960 scope.go:122] "RemoveContainer" containerID="1550b2b3bbdb3edfbe8179b850ed2046b1acd9b3af627808e2e841ffafb01e9a" Oct 30 18:43:19.305516 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:19.305050 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7a8fa2-59479c7948-wtw6n" event={"ID":"da7fcb8d-db2d-4b43-a04f-22f4a9c0757b","Type":"ContainerStarted","Data":"4e52eb5020a77f2e16766747cc8c791fe7c159b0e3542ce3450bac64cf367ff0"} Oct 30 18:43:19.305516 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:19.305115 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7a8fa2-59479c7948-wtw6n" Oct 30 18:43:19.312747 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:19.312730 6960 scope.go:122] "RemoveContainer" containerID="1550b2b3bbdb3edfbe8179b850ed2046b1acd9b3af627808e2e841ffafb01e9a" Oct 30 18:43:19.313038 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:43:19.313016 6960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1550b2b3bbdb3edfbe8179b850ed2046b1acd9b3af627808e2e841ffafb01e9a\": container with ID starting with 1550b2b3bbdb3edfbe8179b850ed2046b1acd9b3af627808e2e841ffafb01e9a not found: ID does not exist" containerID="1550b2b3bbdb3edfbe8179b850ed2046b1acd9b3af627808e2e841ffafb01e9a" Oct 30 18:43:19.313117 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:19.313047 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1550b2b3bbdb3edfbe8179b850ed2046b1acd9b3af627808e2e841ffafb01e9a"} err="failed to get container status \"1550b2b3bbdb3edfbe8179b850ed2046b1acd9b3af627808e2e841ffafb01e9a\": rpc error: code = NotFound desc = could not find container \"1550b2b3bbdb3edfbe8179b850ed2046b1acd9b3af627808e2e841ffafb01e9a\": container with ID starting with 1550b2b3bbdb3edfbe8179b850ed2046b1acd9b3af627808e2e841ffafb01e9a not found: ID does not exist" Oct 30 18:43:19.343449 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:19.343403 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7a8fa2-59479c7948-wtw6n" podStartSLOduration=1.929659118 podStartE2EDuration="2.343389719s" totalImagesPullingTime="413.730601ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:43:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:43:19.341115875 +0000 UTC m=+566.870147009" watchObservedRunningTime="2026-10-30 18:43:19.343389719 +0000 UTC m=+566.872420857" Oct 30 18:43:19.369718 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:19.369687 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-72bc98fe-7ddbc86959-p6tch"] Oct 30 18:43:19.373479 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:19.373443 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-72bc98fe-7ddbc86959-p6tch"] Oct 30 18:43:19.385465 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:19.385441 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:19.509617 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:19.509582 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:19.840899 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:19.840869 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-da70a5-5cb5894d96-ktj45"] Oct 30 18:43:19.841385 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:19.841371 6960 cpu_manager.go:414] "containerMap: removing container" podUID="b64ba209-e869-4bb6-abdd-db517312ea7f" containerName="maas-api" Oct 30 18:43:19.841440 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:19.841386 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b64ba209-e869-4bb6-abdd-db517312ea7f" containerName="maas-api" Oct 30 18:43:19.841476 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:19.841469 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="b64ba209-e869-4bb6-abdd-db517312ea7f" containerName="maas-api" Oct 30 18:43:19.847855 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:19.847836 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-da70a5-5cb5894d96-ktj45" Oct 30 18:43:19.855579 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:19.854900 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-da70a5-serving-cert\"" Oct 30 18:43:19.855579 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:19.855248 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtqmp\" (UniqueName: \"kubernetes.io/projected/5dc3436e-00d1-4ac1-9ea4-feaa1d92a15f-kube-api-access-rtqmp\") pod \"maas-api-e2e-worker-w4-da70a5-5cb5894d96-ktj45\" (UID: \"5dc3436e-00d1-4ac1-9ea4-feaa1d92a15f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-da70a5-5cb5894d96-ktj45" Oct 30 18:43:19.855579 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:19.855295 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dc3436e-00d1-4ac1-9ea4-feaa1d92a15f-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-da70a5-5cb5894d96-ktj45\" (UID: \"5dc3436e-00d1-4ac1-9ea4-feaa1d92a15f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-da70a5-5cb5894d96-ktj45" Oct 30 18:43:19.855579 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:19.855373 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5dc3436e-00d1-4ac1-9ea4-feaa1d92a15f-metrics-certs\") pod \"maas-api-e2e-worker-w4-da70a5-5cb5894d96-ktj45\" (UID: \"5dc3436e-00d1-4ac1-9ea4-feaa1d92a15f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-da70a5-5cb5894d96-ktj45" Oct 30 18:43:19.855579 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:19.855449 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5dc3436e-00d1-4ac1-9ea4-feaa1d92a15f-maas-api-tls\") pod \"maas-api-e2e-worker-w4-da70a5-5cb5894d96-ktj45\" (UID: \"5dc3436e-00d1-4ac1-9ea4-feaa1d92a15f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-da70a5-5cb5894d96-ktj45" Oct 30 18:43:19.858200 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:19.858177 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-da70a5-5cb5894d96-ktj45"] Oct 30 18:43:19.901798 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:19.901762 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:19.956237 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:19.956206 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5dc3436e-00d1-4ac1-9ea4-feaa1d92a15f-maas-api-tls\") pod \"maas-api-e2e-worker-w4-da70a5-5cb5894d96-ktj45\" (UID: \"5dc3436e-00d1-4ac1-9ea4-feaa1d92a15f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-da70a5-5cb5894d96-ktj45" Oct 30 18:43:19.956415 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:19.956305 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtqmp\" (UniqueName: \"kubernetes.io/projected/5dc3436e-00d1-4ac1-9ea4-feaa1d92a15f-kube-api-access-rtqmp\") pod \"maas-api-e2e-worker-w4-da70a5-5cb5894d96-ktj45\" (UID: \"5dc3436e-00d1-4ac1-9ea4-feaa1d92a15f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-da70a5-5cb5894d96-ktj45" Oct 30 18:43:19.956415 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:19.956350 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dc3436e-00d1-4ac1-9ea4-feaa1d92a15f-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-da70a5-5cb5894d96-ktj45\" (UID: \"5dc3436e-00d1-4ac1-9ea4-feaa1d92a15f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-da70a5-5cb5894d96-ktj45" Oct 30 18:43:19.956415 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:43:19.956384 6960 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-da70a5-serving-cert: secret "maas-api-e2e-worker-w4-da70a5-serving-cert" not found Oct 30 18:43:19.956537 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:19.956414 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5dc3436e-00d1-4ac1-9ea4-feaa1d92a15f-metrics-certs\") pod \"maas-api-e2e-worker-w4-da70a5-5cb5894d96-ktj45\" (UID: \"5dc3436e-00d1-4ac1-9ea4-feaa1d92a15f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-da70a5-5cb5894d96-ktj45" Oct 30 18:43:19.956537 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:43:19.956460 6960 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5dc3436e-00d1-4ac1-9ea4-feaa1d92a15f-maas-api-tls podName:5dc3436e-00d1-4ac1-9ea4-feaa1d92a15f nodeName:}" failed. No retries permitted until 2026-10-30 18:43:20.456438581 +0000 UTC m=+567.985469711 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5dc3436e-00d1-4ac1-9ea4-feaa1d92a15f-maas-api-tls") pod "maas-api-e2e-worker-w4-da70a5-5cb5894d96-ktj45" (UID: "5dc3436e-00d1-4ac1-9ea4-feaa1d92a15f") : secret "maas-api-e2e-worker-w4-da70a5-serving-cert" not found Oct 30 18:43:19.957517 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:19.957489 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dc3436e-00d1-4ac1-9ea4-feaa1d92a15f-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-da70a5-5cb5894d96-ktj45\" (UID: \"5dc3436e-00d1-4ac1-9ea4-feaa1d92a15f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-da70a5-5cb5894d96-ktj45" Oct 30 18:43:19.960064 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:19.960037 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5dc3436e-00d1-4ac1-9ea4-feaa1d92a15f-metrics-certs\") pod \"maas-api-e2e-worker-w4-da70a5-5cb5894d96-ktj45\" (UID: \"5dc3436e-00d1-4ac1-9ea4-feaa1d92a15f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-da70a5-5cb5894d96-ktj45" Oct 30 18:43:19.969869 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:19.969836 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtqmp\" (UniqueName: \"kubernetes.io/projected/5dc3436e-00d1-4ac1-9ea4-feaa1d92a15f-kube-api-access-rtqmp\") pod \"maas-api-e2e-worker-w4-da70a5-5cb5894d96-ktj45\" (UID: \"5dc3436e-00d1-4ac1-9ea4-feaa1d92a15f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-da70a5-5cb5894d96-ktj45" Oct 30 18:43:20.015510 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:20.015480 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:20.163333 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:20.163252 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:20.272119 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:20.272088 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:20.460992 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:20.460903 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5dc3436e-00d1-4ac1-9ea4-feaa1d92a15f-maas-api-tls\") pod \"maas-api-e2e-worker-w4-da70a5-5cb5894d96-ktj45\" (UID: \"5dc3436e-00d1-4ac1-9ea4-feaa1d92a15f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-da70a5-5cb5894d96-ktj45" Oct 30 18:43:20.463457 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:20.463436 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5dc3436e-00d1-4ac1-9ea4-feaa1d92a15f-maas-api-tls\") pod \"maas-api-e2e-worker-w4-da70a5-5cb5894d96-ktj45\" (UID: \"5dc3436e-00d1-4ac1-9ea4-feaa1d92a15f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-da70a5-5cb5894d96-ktj45" Oct 30 18:43:20.474458 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:20.474427 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-da70a5-5cb5894d96-ktj45" Oct 30 18:43:20.596243 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:20.596192 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b64ba209-e869-4bb6-abdd-db517312ea7f" path="/var/lib/kubelet/pods/b64ba209-e869-4bb6-abdd-db517312ea7f/volumes" Oct 30 18:43:20.597634 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:20.597604 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:20.760489 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:20.760406 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:20.765215 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:20.765196 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-da70a5-5cb5894d96-ktj45"] Oct 30 18:43:20.767670 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:43:20.767641 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dc3436e_00d1_4ac1_9ea4_feaa1d92a15f.slice/crio-1205646afe42078afff642c3caa1dfa195382bcab55e0b4e90620d80d54f60ef WatchSource:0}: Error finding container 1205646afe42078afff642c3caa1dfa195382bcab55e0b4e90620d80d54f60ef: Status 404 returned error can't find the container with id 1205646afe42078afff642c3caa1dfa195382bcab55e0b4e90620d80d54f60ef Oct 30 18:43:21.318229 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:21.318196 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-da70a5-5cb5894d96-ktj45" event={"ID":"5dc3436e-00d1-4ac1-9ea4-feaa1d92a15f","Type":"ContainerStarted","Data":"c810cb627ec671d1a208549ee2722ce552d182c28c53b2557d424b0b970cf0c8"} Oct 30 18:43:21.318386 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:21.318281 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-da70a5-5cb5894d96-ktj45" event={"ID":"5dc3436e-00d1-4ac1-9ea4-feaa1d92a15f","Type":"ContainerStarted","Data":"1205646afe42078afff642c3caa1dfa195382bcab55e0b4e90620d80d54f60ef"} Oct 30 18:43:21.318386 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:21.318307 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-da70a5-5cb5894d96-ktj45" Oct 30 18:43:21.348067 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:21.348014 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-da70a5-5cb5894d96-ktj45" podStartSLOduration=1.9304016449999999 podStartE2EDuration="2.34799692s" totalImagesPullingTime="417.595275ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:43:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:43:21.347460639 +0000 UTC m=+568.876491778" watchObservedRunningTime="2026-10-30 18:43:21.34799692 +0000 UTC m=+568.877028061" Oct 30 18:43:21.467685 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:21.467568 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:21.742376 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:21.742295 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:21.883833 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:21.883796 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:22.361995 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:22.361966 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:22.501675 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:22.501644 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:22.659010 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:22.658929 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:22.767722 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:22.767693 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:23.085320 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:23.085279 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:23.839625 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:23.839597 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-8a92cc-788c8c9c4b-6gtz2"] Oct 30 18:43:23.843691 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:23.843667 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8a92cc-788c8c9c4b-6gtz2" Oct 30 18:43:23.858592 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:23.858571 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-8a92cc-serving-cert\"" Oct 30 18:43:23.859250 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:23.859229 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-8a92cc-788c8c9c4b-6gtz2"] Oct 30 18:43:23.902920 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:23.902896 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7998c864-d4ca-4b59-9ef6-0838726a0c8c-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-8a92cc-788c8c9c4b-6gtz2\" (UID: \"7998c864-d4ca-4b59-9ef6-0838726a0c8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8a92cc-788c8c9c4b-6gtz2" Oct 30 18:43:23.903062 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:23.902940 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7998c864-d4ca-4b59-9ef6-0838726a0c8c-metrics-certs\") pod \"maas-api-e2e-worker-w2-8a92cc-788c8c9c4b-6gtz2\" (UID: \"7998c864-d4ca-4b59-9ef6-0838726a0c8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8a92cc-788c8c9c4b-6gtz2" Oct 30 18:43:23.903136 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:23.903096 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7998c864-d4ca-4b59-9ef6-0838726a0c8c-maas-api-tls\") pod \"maas-api-e2e-worker-w2-8a92cc-788c8c9c4b-6gtz2\" (UID: \"7998c864-d4ca-4b59-9ef6-0838726a0c8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8a92cc-788c8c9c4b-6gtz2" Oct 30 18:43:23.903311 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:23.903294 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggls4\" (UniqueName: \"kubernetes.io/projected/7998c864-d4ca-4b59-9ef6-0838726a0c8c-kube-api-access-ggls4\") pod \"maas-api-e2e-worker-w2-8a92cc-788c8c9c4b-6gtz2\" (UID: \"7998c864-d4ca-4b59-9ef6-0838726a0c8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8a92cc-788c8c9c4b-6gtz2" Oct 30 18:43:23.930894 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:23.930830 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:23.932045 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:43:23.932018 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 18:43:24.004937 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:24.004905 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggls4\" (UniqueName: \"kubernetes.io/projected/7998c864-d4ca-4b59-9ef6-0838726a0c8c-kube-api-access-ggls4\") pod \"maas-api-e2e-worker-w2-8a92cc-788c8c9c4b-6gtz2\" (UID: \"7998c864-d4ca-4b59-9ef6-0838726a0c8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8a92cc-788c8c9c4b-6gtz2" Oct 30 18:43:24.005070 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:24.004969 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7998c864-d4ca-4b59-9ef6-0838726a0c8c-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-8a92cc-788c8c9c4b-6gtz2\" (UID: \"7998c864-d4ca-4b59-9ef6-0838726a0c8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8a92cc-788c8c9c4b-6gtz2" Oct 30 18:43:24.005070 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:24.005003 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7998c864-d4ca-4b59-9ef6-0838726a0c8c-metrics-certs\") pod \"maas-api-e2e-worker-w2-8a92cc-788c8c9c4b-6gtz2\" (UID: \"7998c864-d4ca-4b59-9ef6-0838726a0c8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8a92cc-788c8c9c4b-6gtz2" Oct 30 18:43:24.005070 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:24.005066 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7998c864-d4ca-4b59-9ef6-0838726a0c8c-maas-api-tls\") pod \"maas-api-e2e-worker-w2-8a92cc-788c8c9c4b-6gtz2\" (UID: \"7998c864-d4ca-4b59-9ef6-0838726a0c8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8a92cc-788c8c9c4b-6gtz2" Oct 30 18:43:24.005621 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:24.005599 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7998c864-d4ca-4b59-9ef6-0838726a0c8c-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-8a92cc-788c8c9c4b-6gtz2\" (UID: \"7998c864-d4ca-4b59-9ef6-0838726a0c8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8a92cc-788c8c9c4b-6gtz2" Oct 30 18:43:24.007554 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:24.007532 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7998c864-d4ca-4b59-9ef6-0838726a0c8c-maas-api-tls\") pod \"maas-api-e2e-worker-w2-8a92cc-788c8c9c4b-6gtz2\" (UID: \"7998c864-d4ca-4b59-9ef6-0838726a0c8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8a92cc-788c8c9c4b-6gtz2" Oct 30 18:43:24.007666 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:24.007595 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7998c864-d4ca-4b59-9ef6-0838726a0c8c-metrics-certs\") pod \"maas-api-e2e-worker-w2-8a92cc-788c8c9c4b-6gtz2\" (UID: \"7998c864-d4ca-4b59-9ef6-0838726a0c8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8a92cc-788c8c9c4b-6gtz2" Oct 30 18:43:24.017518 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:24.017494 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggls4\" (UniqueName: \"kubernetes.io/projected/7998c864-d4ca-4b59-9ef6-0838726a0c8c-kube-api-access-ggls4\") pod \"maas-api-e2e-worker-w2-8a92cc-788c8c9c4b-6gtz2\" (UID: \"7998c864-d4ca-4b59-9ef6-0838726a0c8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8a92cc-788c8c9c4b-6gtz2" Oct 30 18:43:24.035447 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:24.035412 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:24.143762 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:24.143663 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:24.158111 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:24.158083 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8a92cc-788c8c9c4b-6gtz2" Oct 30 18:43:24.451639 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:24.451611 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-8a92cc-788c8c9c4b-6gtz2"] Oct 30 18:43:24.453912 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:43:24.453889 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7998c864_d4ca_4b59_9ef6_0838726a0c8c.slice/crio-c33340675be6901d90c104660b69fa5c59e5a32f230341166c920303814cc9d9 WatchSource:0}: Error finding container c33340675be6901d90c104660b69fa5c59e5a32f230341166c920303814cc9d9: Status 404 returned error can't find the container with id c33340675be6901d90c104660b69fa5c59e5a32f230341166c920303814cc9d9 Oct 30 18:43:24.500897 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:24.500863 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:24.611277 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:24.611244 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:24.919854 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:24.919819 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:25.035172 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:25.035120 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:25.176956 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:25.176865 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:25.317634 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:25.317608 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7a8fa2-59479c7948-wtw6n" Oct 30 18:43:25.338933 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:25.338900 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8a92cc-788c8c9c4b-6gtz2" event={"ID":"7998c864-d4ca-4b59-9ef6-0838726a0c8c","Type":"ContainerStarted","Data":"90dbdc8f6ec52b6ee54d815672417459eb3ecc43e61b3c33fb9dd0135428c6a8"} Oct 30 18:43:25.339105 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:25.338964 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8a92cc-788c8c9c4b-6gtz2" Oct 30 18:43:25.339105 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:25.338985 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8a92cc-788c8c9c4b-6gtz2" event={"ID":"7998c864-d4ca-4b59-9ef6-0838726a0c8c","Type":"ContainerStarted","Data":"c33340675be6901d90c104660b69fa5c59e5a32f230341166c920303814cc9d9"} Oct 30 18:43:25.367320 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:25.367261 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8a92cc-788c8c9c4b-6gtz2" podStartSLOduration=1.955205318 podStartE2EDuration="2.367242293s" totalImagesPullingTime="412.036975ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:43:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:43:25.365801739 +0000 UTC m=+572.894832878" watchObservedRunningTime="2026-10-30 18:43:25.367242293 +0000 UTC m=+572.896273431" Oct 30 18:43:25.816295 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:25.816259 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:26.753867 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:26.753827 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:27.328697 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:27.328666 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-da70a5-5cb5894d96-ktj45" Oct 30 18:43:27.721069 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:27.720968 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:28.298068 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:28.298037 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-7a8fa2-778fb96bb-tlvqh" Oct 30 18:43:28.455880 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:28.455845 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:28.760258 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:28.760219 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:29.042871 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:29.042783 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:29.752880 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:29.752845 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:29.954330 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:29.954297 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:30.110402 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:30.110370 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:30.407682 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:30.407595 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:30.884783 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:30.884750 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:31.066167 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:31.066112 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:31.349516 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:31.349492 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8a92cc-788c8c9c4b-6gtz2" Oct 30 18:43:31.825312 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:31.825280 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:32.966982 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:32.966945 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:33.572274 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:33.572243 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-7a8fa2/simulator-w3-7a8fa2-kserve-78587cb8fc-6pvsz"] Oct 30 18:43:33.575989 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:33.575972 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-7a8fa2/simulator-w3-7a8fa2-kserve-78587cb8fc-6pvsz" Oct 30 18:43:33.589361 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:33.589340 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-7a8fa2\"/\"kube-root-ca.crt\"" Oct 30 18:43:33.589481 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:33.589442 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-7a8fa2\"/\"openshift-service-ca.crt\"" Oct 30 18:43:33.589874 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:33.589855 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-7a8fa2\"/\"simulator-w3-7a8fa2-kserve-self-signed-certs\"" Oct 30 18:43:33.590043 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:33.590027 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-7a8fa2\"/\"default-dockercfg-f2bkf\"" Oct 30 18:43:33.598453 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:33.598428 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-7a8fa2/simulator-w3-7a8fa2-kserve-78587cb8fc-6pvsz"] Oct 30 18:43:33.644795 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:33.644755 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:33.704554 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:33.704519 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc64ebb9-44e9-435a-9db0-d53ce5f139ff-home\") pod \"simulator-w3-7a8fa2-kserve-78587cb8fc-6pvsz\" (UID: \"cc64ebb9-44e9-435a-9db0-d53ce5f139ff\") " pod="e2e-models-e2e-worker-w3-7a8fa2/simulator-w3-7a8fa2-kserve-78587cb8fc-6pvsz" Oct 30 18:43:33.704764 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:33.704574 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc64ebb9-44e9-435a-9db0-d53ce5f139ff-model-cache\") pod \"simulator-w3-7a8fa2-kserve-78587cb8fc-6pvsz\" (UID: \"cc64ebb9-44e9-435a-9db0-d53ce5f139ff\") " pod="e2e-models-e2e-worker-w3-7a8fa2/simulator-w3-7a8fa2-kserve-78587cb8fc-6pvsz" Oct 30 18:43:33.704764 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:33.704644 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc64ebb9-44e9-435a-9db0-d53ce5f139ff-tmp-dir\") pod \"simulator-w3-7a8fa2-kserve-78587cb8fc-6pvsz\" (UID: \"cc64ebb9-44e9-435a-9db0-d53ce5f139ff\") " pod="e2e-models-e2e-worker-w3-7a8fa2/simulator-w3-7a8fa2-kserve-78587cb8fc-6pvsz" Oct 30 18:43:33.704764 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:33.704698 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lg4ll\" (UniqueName: \"kubernetes.io/projected/cc64ebb9-44e9-435a-9db0-d53ce5f139ff-kube-api-access-lg4ll\") pod \"simulator-w3-7a8fa2-kserve-78587cb8fc-6pvsz\" (UID: \"cc64ebb9-44e9-435a-9db0-d53ce5f139ff\") " pod="e2e-models-e2e-worker-w3-7a8fa2/simulator-w3-7a8fa2-kserve-78587cb8fc-6pvsz" Oct 30 18:43:33.704868 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:33.704813 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc64ebb9-44e9-435a-9db0-d53ce5f139ff-tls-certs\") pod \"simulator-w3-7a8fa2-kserve-78587cb8fc-6pvsz\" (UID: \"cc64ebb9-44e9-435a-9db0-d53ce5f139ff\") " pod="e2e-models-e2e-worker-w3-7a8fa2/simulator-w3-7a8fa2-kserve-78587cb8fc-6pvsz" Oct 30 18:43:33.704868 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:33.704850 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc64ebb9-44e9-435a-9db0-d53ce5f139ff-dshm\") pod \"simulator-w3-7a8fa2-kserve-78587cb8fc-6pvsz\" (UID: \"cc64ebb9-44e9-435a-9db0-d53ce5f139ff\") " pod="e2e-models-e2e-worker-w3-7a8fa2/simulator-w3-7a8fa2-kserve-78587cb8fc-6pvsz" Oct 30 18:43:33.805701 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:33.805648 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc64ebb9-44e9-435a-9db0-d53ce5f139ff-tls-certs\") pod \"simulator-w3-7a8fa2-kserve-78587cb8fc-6pvsz\" (UID: \"cc64ebb9-44e9-435a-9db0-d53ce5f139ff\") " pod="e2e-models-e2e-worker-w3-7a8fa2/simulator-w3-7a8fa2-kserve-78587cb8fc-6pvsz" Oct 30 18:43:33.805904 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:33.805765 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc64ebb9-44e9-435a-9db0-d53ce5f139ff-dshm\") pod \"simulator-w3-7a8fa2-kserve-78587cb8fc-6pvsz\" (UID: \"cc64ebb9-44e9-435a-9db0-d53ce5f139ff\") " pod="e2e-models-e2e-worker-w3-7a8fa2/simulator-w3-7a8fa2-kserve-78587cb8fc-6pvsz" Oct 30 18:43:33.805904 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:33.805825 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc64ebb9-44e9-435a-9db0-d53ce5f139ff-home\") pod \"simulator-w3-7a8fa2-kserve-78587cb8fc-6pvsz\" (UID: \"cc64ebb9-44e9-435a-9db0-d53ce5f139ff\") " pod="e2e-models-e2e-worker-w3-7a8fa2/simulator-w3-7a8fa2-kserve-78587cb8fc-6pvsz" Oct 30 18:43:33.805904 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:33.805891 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc64ebb9-44e9-435a-9db0-d53ce5f139ff-model-cache\") pod \"simulator-w3-7a8fa2-kserve-78587cb8fc-6pvsz\" (UID: \"cc64ebb9-44e9-435a-9db0-d53ce5f139ff\") " pod="e2e-models-e2e-worker-w3-7a8fa2/simulator-w3-7a8fa2-kserve-78587cb8fc-6pvsz" Oct 30 18:43:33.806099 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:33.805920 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc64ebb9-44e9-435a-9db0-d53ce5f139ff-tmp-dir\") pod \"simulator-w3-7a8fa2-kserve-78587cb8fc-6pvsz\" (UID: \"cc64ebb9-44e9-435a-9db0-d53ce5f139ff\") " pod="e2e-models-e2e-worker-w3-7a8fa2/simulator-w3-7a8fa2-kserve-78587cb8fc-6pvsz" Oct 30 18:43:33.806099 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:33.805985 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lg4ll\" (UniqueName: \"kubernetes.io/projected/cc64ebb9-44e9-435a-9db0-d53ce5f139ff-kube-api-access-lg4ll\") pod \"simulator-w3-7a8fa2-kserve-78587cb8fc-6pvsz\" (UID: \"cc64ebb9-44e9-435a-9db0-d53ce5f139ff\") " pod="e2e-models-e2e-worker-w3-7a8fa2/simulator-w3-7a8fa2-kserve-78587cb8fc-6pvsz" Oct 30 18:43:33.806362 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:33.806334 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc64ebb9-44e9-435a-9db0-d53ce5f139ff-model-cache\") pod \"simulator-w3-7a8fa2-kserve-78587cb8fc-6pvsz\" (UID: \"cc64ebb9-44e9-435a-9db0-d53ce5f139ff\") " pod="e2e-models-e2e-worker-w3-7a8fa2/simulator-w3-7a8fa2-kserve-78587cb8fc-6pvsz" Oct 30 18:43:33.806362 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:33.806353 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc64ebb9-44e9-435a-9db0-d53ce5f139ff-home\") pod \"simulator-w3-7a8fa2-kserve-78587cb8fc-6pvsz\" (UID: \"cc64ebb9-44e9-435a-9db0-d53ce5f139ff\") " pod="e2e-models-e2e-worker-w3-7a8fa2/simulator-w3-7a8fa2-kserve-78587cb8fc-6pvsz" Oct 30 18:43:33.806535 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:33.806465 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc64ebb9-44e9-435a-9db0-d53ce5f139ff-tmp-dir\") pod \"simulator-w3-7a8fa2-kserve-78587cb8fc-6pvsz\" (UID: \"cc64ebb9-44e9-435a-9db0-d53ce5f139ff\") " pod="e2e-models-e2e-worker-w3-7a8fa2/simulator-w3-7a8fa2-kserve-78587cb8fc-6pvsz" Oct 30 18:43:33.808377 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:33.808356 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc64ebb9-44e9-435a-9db0-d53ce5f139ff-dshm\") pod \"simulator-w3-7a8fa2-kserve-78587cb8fc-6pvsz\" (UID: \"cc64ebb9-44e9-435a-9db0-d53ce5f139ff\") " pod="e2e-models-e2e-worker-w3-7a8fa2/simulator-w3-7a8fa2-kserve-78587cb8fc-6pvsz" Oct 30 18:43:33.808648 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:33.808628 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc64ebb9-44e9-435a-9db0-d53ce5f139ff-tls-certs\") pod \"simulator-w3-7a8fa2-kserve-78587cb8fc-6pvsz\" (UID: \"cc64ebb9-44e9-435a-9db0-d53ce5f139ff\") " pod="e2e-models-e2e-worker-w3-7a8fa2/simulator-w3-7a8fa2-kserve-78587cb8fc-6pvsz" Oct 30 18:43:33.818327 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:33.818310 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lg4ll\" (UniqueName: \"kubernetes.io/projected/cc64ebb9-44e9-435a-9db0-d53ce5f139ff-kube-api-access-lg4ll\") pod \"simulator-w3-7a8fa2-kserve-78587cb8fc-6pvsz\" (UID: \"cc64ebb9-44e9-435a-9db0-d53ce5f139ff\") " pod="e2e-models-e2e-worker-w3-7a8fa2/simulator-w3-7a8fa2-kserve-78587cb8fc-6pvsz" Oct 30 18:43:33.891538 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:33.891471 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-7a8fa2/simulator-w3-7a8fa2-kserve-78587cb8fc-6pvsz" Oct 30 18:43:33.934768 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:33.934739 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:34.069637 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:34.069605 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:34.181365 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:34.181121 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-7a8fa2/simulator-w3-7a8fa2-kserve-78587cb8fc-6pvsz"] Oct 30 18:43:34.183021 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:43:34.182995 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcc64ebb9_44e9_435a_9db0_d53ce5f139ff.slice/crio-1cf18d107341fbfd9bf8a254030a6afa1d5c4b0e014cee0e391f2cc909e522e5 WatchSource:0}: Error finding container 1cf18d107341fbfd9bf8a254030a6afa1d5c4b0e014cee0e391f2cc909e522e5: Status 404 returned error can't find the container with id 1cf18d107341fbfd9bf8a254030a6afa1d5c4b0e014cee0e391f2cc909e522e5 Oct 30 18:43:34.222013 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:34.221985 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:34.389761 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:34.389729 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-7a8fa2/simulator-w3-7a8fa2-kserve-78587cb8fc-6pvsz" event={"ID":"cc64ebb9-44e9-435a-9db0-d53ce5f139ff","Type":"ContainerStarted","Data":"1cf18d107341fbfd9bf8a254030a6afa1d5c4b0e014cee0e391f2cc909e522e5"} Oct 30 18:43:35.027304 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:35.027264 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:36.190256 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:36.190224 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:36.402333 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:36.402294 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-7a8fa2/simulator-w3-7a8fa2-kserve-78587cb8fc-6pvsz" event={"ID":"cc64ebb9-44e9-435a-9db0-d53ce5f139ff","Type":"ContainerStarted","Data":"1971b54e599f3ec447c9fc92d5fa8bde1edea9aebb77a9805e45805e6765d9d7"} Oct 30 18:43:36.437774 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:36.437727 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-7a8fa2/simulator-w3-7a8fa2-kserve-78587cb8fc-6pvsz" podStartSLOduration=2.117826363 podStartE2EDuration="3.437713887s" totalImagesPullingTime="1.319887524s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:43:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:43:36.43621568 +0000 UTC m=+583.965246817" watchObservedRunningTime="2026-10-30 18:43:36.437713887 +0000 UTC m=+583.966745028" Oct 30 18:43:36.892164 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:36.892115 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7a8fa2/simulator-w3-7a8fa2-kserve-78587cb8fc-6pvsz" Oct 30 18:43:37.119985 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:37.119941 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:38.316031 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:38.315999 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:38.430348 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:38.430319 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:39.038282 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:39.038249 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:39.154130 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:39.154098 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:39.272845 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:39.272803 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:40.220916 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:40.220879 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:40.333737 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:40.333707 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:40.458455 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:40.458420 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:40.567534 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:40.567499 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:41.005603 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:41.005527 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:41.234986 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:41.234942 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:41.426852 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:41.426796 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:41.579966 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:41.579930 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:42.749284 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:42.749235 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:43.561658 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:43.561614 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:43.891997 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:43.891919 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-7a8fa2/simulator-w3-7a8fa2-kserve-78587cb8fc-6pvsz" Oct 30 18:43:43.904839 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:43.904817 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-7a8fa2/simulator-w3-7a8fa2-kserve-78587cb8fc-6pvsz" Oct 30 18:43:44.387989 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:44.387950 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:44.455576 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:44.455546 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7a8fa2/simulator-w3-7a8fa2-kserve-78587cb8fc-6pvsz" Oct 30 18:43:45.250134 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:45.250095 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:46.235414 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:46.235376 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:47.344098 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:47.344058 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:48.344438 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:48.344392 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:48.611639 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:48.611541 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:49.347899 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:49.347863 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:49.526916 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:49.526876 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:50.129736 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:50.129709 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-8a92cc/simulator-w2-8a92cc-kserve-fb77d84bc-nqf9q"] Oct 30 18:43:50.133852 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:50.133829 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-8a92cc/simulator-w2-8a92cc-kserve-fb77d84bc-nqf9q" Oct 30 18:43:50.178880 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:50.178851 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-8a92cc/simulator-w2-8a92cc-kserve-fb77d84bc-nqf9q"] Oct 30 18:43:50.186274 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:50.186249 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-8a92cc\"/\"kube-root-ca.crt\"" Oct 30 18:43:50.186397 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:50.186344 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-8a92cc\"/\"openshift-service-ca.crt\"" Oct 30 18:43:50.219274 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:50.219246 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-8a92cc\"/\"simulator-w2-8a92cc-kserve-self-signed-certs\"" Oct 30 18:43:50.219274 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:50.219249 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-8a92cc\"/\"default-dockercfg-ttpl9\"" Oct 30 18:43:50.275503 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:50.275461 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce70492e-c727-4dc1-b2ee-bf278d29a45b-home\") pod \"simulator-w2-8a92cc-kserve-fb77d84bc-nqf9q\" (UID: \"ce70492e-c727-4dc1-b2ee-bf278d29a45b\") " pod="e2e-models-e2e-worker-w2-8a92cc/simulator-w2-8a92cc-kserve-fb77d84bc-nqf9q" Oct 30 18:43:50.275697 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:50.275526 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce70492e-c727-4dc1-b2ee-bf278d29a45b-tls-certs\") pod \"simulator-w2-8a92cc-kserve-fb77d84bc-nqf9q\" (UID: \"ce70492e-c727-4dc1-b2ee-bf278d29a45b\") " pod="e2e-models-e2e-worker-w2-8a92cc/simulator-w2-8a92cc-kserve-fb77d84bc-nqf9q" Oct 30 18:43:50.275697 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:50.275675 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce70492e-c727-4dc1-b2ee-bf278d29a45b-tmp-dir\") pod \"simulator-w2-8a92cc-kserve-fb77d84bc-nqf9q\" (UID: \"ce70492e-c727-4dc1-b2ee-bf278d29a45b\") " pod="e2e-models-e2e-worker-w2-8a92cc/simulator-w2-8a92cc-kserve-fb77d84bc-nqf9q" Oct 30 18:43:50.275803 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:50.275783 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce70492e-c727-4dc1-b2ee-bf278d29a45b-dshm\") pod \"simulator-w2-8a92cc-kserve-fb77d84bc-nqf9q\" (UID: \"ce70492e-c727-4dc1-b2ee-bf278d29a45b\") " pod="e2e-models-e2e-worker-w2-8a92cc/simulator-w2-8a92cc-kserve-fb77d84bc-nqf9q" Oct 30 18:43:50.275841 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:50.275831 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce70492e-c727-4dc1-b2ee-bf278d29a45b-model-cache\") pod \"simulator-w2-8a92cc-kserve-fb77d84bc-nqf9q\" (UID: \"ce70492e-c727-4dc1-b2ee-bf278d29a45b\") " pod="e2e-models-e2e-worker-w2-8a92cc/simulator-w2-8a92cc-kserve-fb77d84bc-nqf9q" Oct 30 18:43:50.275874 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:50.275858 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4knng\" (UniqueName: \"kubernetes.io/projected/ce70492e-c727-4dc1-b2ee-bf278d29a45b-kube-api-access-4knng\") pod \"simulator-w2-8a92cc-kserve-fb77d84bc-nqf9q\" (UID: \"ce70492e-c727-4dc1-b2ee-bf278d29a45b\") " pod="e2e-models-e2e-worker-w2-8a92cc/simulator-w2-8a92cc-kserve-fb77d84bc-nqf9q" Oct 30 18:43:50.295347 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:50.295321 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:50.376746 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:50.376719 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce70492e-c727-4dc1-b2ee-bf278d29a45b-home\") pod \"simulator-w2-8a92cc-kserve-fb77d84bc-nqf9q\" (UID: \"ce70492e-c727-4dc1-b2ee-bf278d29a45b\") " pod="e2e-models-e2e-worker-w2-8a92cc/simulator-w2-8a92cc-kserve-fb77d84bc-nqf9q" Oct 30 18:43:50.377133 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:50.376762 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce70492e-c727-4dc1-b2ee-bf278d29a45b-tls-certs\") pod \"simulator-w2-8a92cc-kserve-fb77d84bc-nqf9q\" (UID: \"ce70492e-c727-4dc1-b2ee-bf278d29a45b\") " pod="e2e-models-e2e-worker-w2-8a92cc/simulator-w2-8a92cc-kserve-fb77d84bc-nqf9q" Oct 30 18:43:50.377133 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:50.376857 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce70492e-c727-4dc1-b2ee-bf278d29a45b-tmp-dir\") pod \"simulator-w2-8a92cc-kserve-fb77d84bc-nqf9q\" (UID: \"ce70492e-c727-4dc1-b2ee-bf278d29a45b\") " pod="e2e-models-e2e-worker-w2-8a92cc/simulator-w2-8a92cc-kserve-fb77d84bc-nqf9q" Oct 30 18:43:50.377133 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:50.376929 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce70492e-c727-4dc1-b2ee-bf278d29a45b-dshm\") pod \"simulator-w2-8a92cc-kserve-fb77d84bc-nqf9q\" (UID: \"ce70492e-c727-4dc1-b2ee-bf278d29a45b\") " pod="e2e-models-e2e-worker-w2-8a92cc/simulator-w2-8a92cc-kserve-fb77d84bc-nqf9q" Oct 30 18:43:50.377133 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:50.376968 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce70492e-c727-4dc1-b2ee-bf278d29a45b-model-cache\") pod \"simulator-w2-8a92cc-kserve-fb77d84bc-nqf9q\" (UID: \"ce70492e-c727-4dc1-b2ee-bf278d29a45b\") " pod="e2e-models-e2e-worker-w2-8a92cc/simulator-w2-8a92cc-kserve-fb77d84bc-nqf9q" Oct 30 18:43:50.377133 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:50.377001 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4knng\" (UniqueName: \"kubernetes.io/projected/ce70492e-c727-4dc1-b2ee-bf278d29a45b-kube-api-access-4knng\") pod \"simulator-w2-8a92cc-kserve-fb77d84bc-nqf9q\" (UID: \"ce70492e-c727-4dc1-b2ee-bf278d29a45b\") " pod="e2e-models-e2e-worker-w2-8a92cc/simulator-w2-8a92cc-kserve-fb77d84bc-nqf9q" Oct 30 18:43:50.377405 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:50.377182 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce70492e-c727-4dc1-b2ee-bf278d29a45b-home\") pod \"simulator-w2-8a92cc-kserve-fb77d84bc-nqf9q\" (UID: \"ce70492e-c727-4dc1-b2ee-bf278d29a45b\") " pod="e2e-models-e2e-worker-w2-8a92cc/simulator-w2-8a92cc-kserve-fb77d84bc-nqf9q" Oct 30 18:43:50.377405 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:50.377226 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce70492e-c727-4dc1-b2ee-bf278d29a45b-tmp-dir\") pod \"simulator-w2-8a92cc-kserve-fb77d84bc-nqf9q\" (UID: \"ce70492e-c727-4dc1-b2ee-bf278d29a45b\") " pod="e2e-models-e2e-worker-w2-8a92cc/simulator-w2-8a92cc-kserve-fb77d84bc-nqf9q" Oct 30 18:43:50.377405 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:50.377321 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce70492e-c727-4dc1-b2ee-bf278d29a45b-model-cache\") pod \"simulator-w2-8a92cc-kserve-fb77d84bc-nqf9q\" (UID: \"ce70492e-c727-4dc1-b2ee-bf278d29a45b\") " pod="e2e-models-e2e-worker-w2-8a92cc/simulator-w2-8a92cc-kserve-fb77d84bc-nqf9q" Oct 30 18:43:50.379265 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:50.379238 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce70492e-c727-4dc1-b2ee-bf278d29a45b-dshm\") pod \"simulator-w2-8a92cc-kserve-fb77d84bc-nqf9q\" (UID: \"ce70492e-c727-4dc1-b2ee-bf278d29a45b\") " pod="e2e-models-e2e-worker-w2-8a92cc/simulator-w2-8a92cc-kserve-fb77d84bc-nqf9q" Oct 30 18:43:50.379491 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:50.379474 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce70492e-c727-4dc1-b2ee-bf278d29a45b-tls-certs\") pod \"simulator-w2-8a92cc-kserve-fb77d84bc-nqf9q\" (UID: \"ce70492e-c727-4dc1-b2ee-bf278d29a45b\") " pod="e2e-models-e2e-worker-w2-8a92cc/simulator-w2-8a92cc-kserve-fb77d84bc-nqf9q" Oct 30 18:43:50.393023 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:50.392971 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4knng\" (UniqueName: \"kubernetes.io/projected/ce70492e-c727-4dc1-b2ee-bf278d29a45b-kube-api-access-4knng\") pod \"simulator-w2-8a92cc-kserve-fb77d84bc-nqf9q\" (UID: \"ce70492e-c727-4dc1-b2ee-bf278d29a45b\") " pod="e2e-models-e2e-worker-w2-8a92cc/simulator-w2-8a92cc-kserve-fb77d84bc-nqf9q" Oct 30 18:43:50.446246 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:50.446220 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-8a92cc/simulator-w2-8a92cc-kserve-fb77d84bc-nqf9q" Oct 30 18:43:50.454819 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:50.454787 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:50.730346 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:50.730315 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-8a92cc/simulator-w2-8a92cc-kserve-fb77d84bc-nqf9q"] Oct 30 18:43:50.731956 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:43:50.731930 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podce70492e_c727_4dc1_b2ee_bf278d29a45b.slice/crio-081f4df771a95056f467678a17917fedb56dc38bdb907922da70305c1f5d968e WatchSource:0}: Error finding container 081f4df771a95056f467678a17917fedb56dc38bdb907922da70305c1f5d968e: Status 404 returned error can't find the container with id 081f4df771a95056f467678a17917fedb56dc38bdb907922da70305c1f5d968e Oct 30 18:43:51.097087 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:51.097053 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:51.490334 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:51.490246 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-8a92cc/simulator-w2-8a92cc-kserve-fb77d84bc-nqf9q" event={"ID":"ce70492e-c727-4dc1-b2ee-bf278d29a45b","Type":"ContainerStarted","Data":"d0e5564d705a626eda818f4877ab018aa9b256cd9a24f804b3ffad218b6c744a"} Oct 30 18:43:51.490334 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:51.490299 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-8a92cc/simulator-w2-8a92cc-kserve-fb77d84bc-nqf9q" event={"ID":"ce70492e-c727-4dc1-b2ee-bf278d29a45b","Type":"ContainerStarted","Data":"081f4df771a95056f467678a17917fedb56dc38bdb907922da70305c1f5d968e"} Oct 30 18:43:51.538277 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:51.538229 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-8a92cc/simulator-w2-8a92cc-kserve-fb77d84bc-nqf9q" podStartSLOduration=1.538216027 podStartE2EDuration="1.538216027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:43:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:43:51.536589967 +0000 UTC m=+599.065621106" watchObservedRunningTime="2026-10-30 18:43:51.538216027 +0000 UTC m=+599.067247168" Oct 30 18:43:52.446905 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:52.446867 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8a92cc/simulator-w2-8a92cc-kserve-fb77d84bc-nqf9q" Oct 30 18:43:52.629812 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:52.629782 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:52.780790 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:52.780694 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:53.851126 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:53.851095 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:53.932260 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:43:53.932226 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 18:43:54.630025 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:54.629990 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:55.641976 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:55.641943 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:56.732528 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:56.732499 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:57.516373 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:57.516339 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:58.648339 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:58.648301 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:59.021928 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:59.021840 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-aa6736/simulator-w1-aa6736-kserve-76985dcd5b-njvnn"] Oct 30 18:43:59.028119 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:59.028097 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-aa6736/simulator-w1-aa6736-kserve-76985dcd5b-njvnn" Oct 30 18:43:59.037108 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:59.037081 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-aa6736\"/\"simulator-w1-aa6736-kserve-self-signed-certs\"" Oct 30 18:43:59.037108 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:59.037100 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-aa6736\"/\"kube-root-ca.crt\"" Oct 30 18:43:59.037438 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:59.037245 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-aa6736\"/\"default-dockercfg-kzm7f\"" Oct 30 18:43:59.037438 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:59.037283 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-aa6736\"/\"openshift-service-ca.crt\"" Oct 30 18:43:59.039898 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:59.039873 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-aa6736/simulator-w1-aa6736-kserve-76985dcd5b-njvnn"] Oct 30 18:43:59.072339 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:59.072311 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/943f7840-8d54-4845-b7ec-c0da72ba9c67-model-cache\") pod \"simulator-w1-aa6736-kserve-76985dcd5b-njvnn\" (UID: \"943f7840-8d54-4845-b7ec-c0da72ba9c67\") " pod="e2e-models-e2e-worker-w1-aa6736/simulator-w1-aa6736-kserve-76985dcd5b-njvnn" Oct 30 18:43:59.072495 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:59.072348 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/943f7840-8d54-4845-b7ec-c0da72ba9c67-tls-certs\") pod \"simulator-w1-aa6736-kserve-76985dcd5b-njvnn\" (UID: \"943f7840-8d54-4845-b7ec-c0da72ba9c67\") " pod="e2e-models-e2e-worker-w1-aa6736/simulator-w1-aa6736-kserve-76985dcd5b-njvnn" Oct 30 18:43:59.072578 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:59.072528 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/943f7840-8d54-4845-b7ec-c0da72ba9c67-tmp-dir\") pod \"simulator-w1-aa6736-kserve-76985dcd5b-njvnn\" (UID: \"943f7840-8d54-4845-b7ec-c0da72ba9c67\") " pod="e2e-models-e2e-worker-w1-aa6736/simulator-w1-aa6736-kserve-76985dcd5b-njvnn" Oct 30 18:43:59.072638 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:59.072589 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/943f7840-8d54-4845-b7ec-c0da72ba9c67-dshm\") pod \"simulator-w1-aa6736-kserve-76985dcd5b-njvnn\" (UID: \"943f7840-8d54-4845-b7ec-c0da72ba9c67\") " pod="e2e-models-e2e-worker-w1-aa6736/simulator-w1-aa6736-kserve-76985dcd5b-njvnn" Oct 30 18:43:59.072638 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:59.072626 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/943f7840-8d54-4845-b7ec-c0da72ba9c67-home\") pod \"simulator-w1-aa6736-kserve-76985dcd5b-njvnn\" (UID: \"943f7840-8d54-4845-b7ec-c0da72ba9c67\") " pod="e2e-models-e2e-worker-w1-aa6736/simulator-w1-aa6736-kserve-76985dcd5b-njvnn" Oct 30 18:43:59.072727 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:59.072712 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2xhz\" (UniqueName: \"kubernetes.io/projected/943f7840-8d54-4845-b7ec-c0da72ba9c67-kube-api-access-x2xhz\") pod \"simulator-w1-aa6736-kserve-76985dcd5b-njvnn\" (UID: \"943f7840-8d54-4845-b7ec-c0da72ba9c67\") " pod="e2e-models-e2e-worker-w1-aa6736/simulator-w1-aa6736-kserve-76985dcd5b-njvnn" Oct 30 18:43:59.090654 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:59.090621 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:59.173866 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:59.173836 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/943f7840-8d54-4845-b7ec-c0da72ba9c67-tmp-dir\") pod \"simulator-w1-aa6736-kserve-76985dcd5b-njvnn\" (UID: \"943f7840-8d54-4845-b7ec-c0da72ba9c67\") " pod="e2e-models-e2e-worker-w1-aa6736/simulator-w1-aa6736-kserve-76985dcd5b-njvnn" Oct 30 18:43:59.174028 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:59.173880 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/943f7840-8d54-4845-b7ec-c0da72ba9c67-dshm\") pod \"simulator-w1-aa6736-kserve-76985dcd5b-njvnn\" (UID: \"943f7840-8d54-4845-b7ec-c0da72ba9c67\") " pod="e2e-models-e2e-worker-w1-aa6736/simulator-w1-aa6736-kserve-76985dcd5b-njvnn" Oct 30 18:43:59.174028 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:59.173914 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/943f7840-8d54-4845-b7ec-c0da72ba9c67-home\") pod \"simulator-w1-aa6736-kserve-76985dcd5b-njvnn\" (UID: \"943f7840-8d54-4845-b7ec-c0da72ba9c67\") " pod="e2e-models-e2e-worker-w1-aa6736/simulator-w1-aa6736-kserve-76985dcd5b-njvnn" Oct 30 18:43:59.174028 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:59.173975 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2xhz\" (UniqueName: \"kubernetes.io/projected/943f7840-8d54-4845-b7ec-c0da72ba9c67-kube-api-access-x2xhz\") pod \"simulator-w1-aa6736-kserve-76985dcd5b-njvnn\" (UID: \"943f7840-8d54-4845-b7ec-c0da72ba9c67\") " pod="e2e-models-e2e-worker-w1-aa6736/simulator-w1-aa6736-kserve-76985dcd5b-njvnn" Oct 30 18:43:59.174217 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:59.174055 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/943f7840-8d54-4845-b7ec-c0da72ba9c67-model-cache\") pod \"simulator-w1-aa6736-kserve-76985dcd5b-njvnn\" (UID: \"943f7840-8d54-4845-b7ec-c0da72ba9c67\") " pod="e2e-models-e2e-worker-w1-aa6736/simulator-w1-aa6736-kserve-76985dcd5b-njvnn" Oct 30 18:43:59.174217 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:59.174090 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/943f7840-8d54-4845-b7ec-c0da72ba9c67-tls-certs\") pod \"simulator-w1-aa6736-kserve-76985dcd5b-njvnn\" (UID: \"943f7840-8d54-4845-b7ec-c0da72ba9c67\") " pod="e2e-models-e2e-worker-w1-aa6736/simulator-w1-aa6736-kserve-76985dcd5b-njvnn" Oct 30 18:43:59.174348 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:59.174328 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/943f7840-8d54-4845-b7ec-c0da72ba9c67-tmp-dir\") pod \"simulator-w1-aa6736-kserve-76985dcd5b-njvnn\" (UID: \"943f7840-8d54-4845-b7ec-c0da72ba9c67\") " pod="e2e-models-e2e-worker-w1-aa6736/simulator-w1-aa6736-kserve-76985dcd5b-njvnn" Oct 30 18:43:59.174417 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:59.174386 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/943f7840-8d54-4845-b7ec-c0da72ba9c67-home\") pod \"simulator-w1-aa6736-kserve-76985dcd5b-njvnn\" (UID: \"943f7840-8d54-4845-b7ec-c0da72ba9c67\") " pod="e2e-models-e2e-worker-w1-aa6736/simulator-w1-aa6736-kserve-76985dcd5b-njvnn" Oct 30 18:43:59.174508 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:59.174486 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/943f7840-8d54-4845-b7ec-c0da72ba9c67-model-cache\") pod \"simulator-w1-aa6736-kserve-76985dcd5b-njvnn\" (UID: \"943f7840-8d54-4845-b7ec-c0da72ba9c67\") " pod="e2e-models-e2e-worker-w1-aa6736/simulator-w1-aa6736-kserve-76985dcd5b-njvnn" Oct 30 18:43:59.176546 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:59.176520 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/943f7840-8d54-4845-b7ec-c0da72ba9c67-dshm\") pod \"simulator-w1-aa6736-kserve-76985dcd5b-njvnn\" (UID: \"943f7840-8d54-4845-b7ec-c0da72ba9c67\") " pod="e2e-models-e2e-worker-w1-aa6736/simulator-w1-aa6736-kserve-76985dcd5b-njvnn" Oct 30 18:43:59.177196 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:59.177175 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/943f7840-8d54-4845-b7ec-c0da72ba9c67-tls-certs\") pod \"simulator-w1-aa6736-kserve-76985dcd5b-njvnn\" (UID: \"943f7840-8d54-4845-b7ec-c0da72ba9c67\") " pod="e2e-models-e2e-worker-w1-aa6736/simulator-w1-aa6736-kserve-76985dcd5b-njvnn" Oct 30 18:43:59.188290 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:59.188265 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2xhz\" (UniqueName: \"kubernetes.io/projected/943f7840-8d54-4845-b7ec-c0da72ba9c67-kube-api-access-x2xhz\") pod \"simulator-w1-aa6736-kserve-76985dcd5b-njvnn\" (UID: \"943f7840-8d54-4845-b7ec-c0da72ba9c67\") " pod="e2e-models-e2e-worker-w1-aa6736/simulator-w1-aa6736-kserve-76985dcd5b-njvnn" Oct 30 18:43:59.345187 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:59.345131 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-aa6736/simulator-w1-aa6736-kserve-76985dcd5b-njvnn" Oct 30 18:43:59.531538 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:59.531498 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:59.629778 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:43:59.629754 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-aa6736/simulator-w1-aa6736-kserve-76985dcd5b-njvnn"] Oct 30 18:43:59.631942 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:43:59.631917 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod943f7840_8d54_4845_b7ec_c0da72ba9c67.slice/crio-8f239e29fd9f496e358c7e13fee9ad23cf3b89c72830fe898756d3fb76d43055 WatchSource:0}: Error finding container 8f239e29fd9f496e358c7e13fee9ad23cf3b89c72830fe898756d3fb76d43055: Status 404 returned error can't find the container with id 8f239e29fd9f496e358c7e13fee9ad23cf3b89c72830fe898756d3fb76d43055 Oct 30 18:44:00.447376 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:00.447350 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-8a92cc/simulator-w2-8a92cc-kserve-fb77d84bc-nqf9q" Oct 30 18:44:00.459948 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:00.459919 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-8a92cc/simulator-w2-8a92cc-kserve-fb77d84bc-nqf9q" Oct 30 18:44:00.502592 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:00.502558 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:00.540033 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:00.539984 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-aa6736/simulator-w1-aa6736-kserve-76985dcd5b-njvnn" event={"ID":"943f7840-8d54-4845-b7ec-c0da72ba9c67","Type":"ContainerStarted","Data":"af2a3147a3a36da06b44805485145efe0622412f4c5e6c9f94c0b755a6811f2b"} Oct 30 18:44:00.540244 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:00.540041 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-aa6736/simulator-w1-aa6736-kserve-76985dcd5b-njvnn" event={"ID":"943f7840-8d54-4845-b7ec-c0da72ba9c67","Type":"ContainerStarted","Data":"8f239e29fd9f496e358c7e13fee9ad23cf3b89c72830fe898756d3fb76d43055"} Oct 30 18:44:00.553677 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:00.553650 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8a92cc/simulator-w2-8a92cc-kserve-fb77d84bc-nqf9q" Oct 30 18:44:00.562507 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:00.562460 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-aa6736/simulator-w1-aa6736-kserve-76985dcd5b-njvnn" podStartSLOduration=1.5624429069999999 podStartE2EDuration="1.562442907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:43:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:44:00.561203013 +0000 UTC m=+608.090234152" watchObservedRunningTime="2026-10-30 18:44:00.562442907 +0000 UTC m=+608.091474045" Oct 30 18:44:01.320084 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:01.320047 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:01.345695 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:01.345562 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-aa6736/simulator-w1-aa6736-kserve-76985dcd5b-njvnn" Oct 30 18:44:02.236487 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:02.236453 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:03.298932 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:03.298895 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:03.823326 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:03.823287 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:04.166711 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:04.166623 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:05.094329 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:05.094287 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:05.908193 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:05.908121 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:06.719737 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:06.719705 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:07.521505 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:07.521468 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:08.964982 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:08.964948 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:09.345846 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:09.345816 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-aa6736/simulator-w1-aa6736-kserve-76985dcd5b-njvnn" Oct 30 18:44:09.358533 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:09.358510 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-aa6736/simulator-w1-aa6736-kserve-76985dcd5b-njvnn" Oct 30 18:44:09.595196 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:09.595131 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-7a8fa2/premium-w3-7a8fa2-kserve-6b6c5f6fb8-n5wtq"] Oct 30 18:44:09.599696 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:09.599642 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-7a8fa2/premium-w3-7a8fa2-kserve-6b6c5f6fb8-n5wtq" Oct 30 18:44:09.605072 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:09.604780 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-aa6736/simulator-w1-aa6736-kserve-76985dcd5b-njvnn" Oct 30 18:44:09.609864 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:09.609827 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-7a8fa2\"/\"premium-w3-7a8fa2-kserve-self-signed-certs\"" Oct 30 18:44:09.614248 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:09.614227 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-7a8fa2/premium-w3-7a8fa2-kserve-6b6c5f6fb8-n5wtq"] Oct 30 18:44:09.665418 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:09.665374 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:09.687769 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:09.687715 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kl582\" (UniqueName: \"kubernetes.io/projected/f51fd746-cd85-4608-96a6-f54afa0ec34d-kube-api-access-kl582\") pod \"premium-w3-7a8fa2-kserve-6b6c5f6fb8-n5wtq\" (UID: \"f51fd746-cd85-4608-96a6-f54afa0ec34d\") " pod="e2e-models-e2e-worker-w3-7a8fa2/premium-w3-7a8fa2-kserve-6b6c5f6fb8-n5wtq" Oct 30 18:44:09.687971 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:09.687798 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f51fd746-cd85-4608-96a6-f54afa0ec34d-tmp-dir\") pod \"premium-w3-7a8fa2-kserve-6b6c5f6fb8-n5wtq\" (UID: \"f51fd746-cd85-4608-96a6-f54afa0ec34d\") " pod="e2e-models-e2e-worker-w3-7a8fa2/premium-w3-7a8fa2-kserve-6b6c5f6fb8-n5wtq" Oct 30 18:44:09.687971 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:09.687843 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f51fd746-cd85-4608-96a6-f54afa0ec34d-model-cache\") pod \"premium-w3-7a8fa2-kserve-6b6c5f6fb8-n5wtq\" (UID: \"f51fd746-cd85-4608-96a6-f54afa0ec34d\") " pod="e2e-models-e2e-worker-w3-7a8fa2/premium-w3-7a8fa2-kserve-6b6c5f6fb8-n5wtq" Oct 30 18:44:09.688092 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:09.688024 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f51fd746-cd85-4608-96a6-f54afa0ec34d-dshm\") pod \"premium-w3-7a8fa2-kserve-6b6c5f6fb8-n5wtq\" (UID: \"f51fd746-cd85-4608-96a6-f54afa0ec34d\") " pod="e2e-models-e2e-worker-w3-7a8fa2/premium-w3-7a8fa2-kserve-6b6c5f6fb8-n5wtq" Oct 30 18:44:09.688092 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:09.688064 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f51fd746-cd85-4608-96a6-f54afa0ec34d-home\") pod \"premium-w3-7a8fa2-kserve-6b6c5f6fb8-n5wtq\" (UID: \"f51fd746-cd85-4608-96a6-f54afa0ec34d\") " pod="e2e-models-e2e-worker-w3-7a8fa2/premium-w3-7a8fa2-kserve-6b6c5f6fb8-n5wtq" Oct 30 18:44:09.688242 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:09.688180 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f51fd746-cd85-4608-96a6-f54afa0ec34d-tls-certs\") pod \"premium-w3-7a8fa2-kserve-6b6c5f6fb8-n5wtq\" (UID: \"f51fd746-cd85-4608-96a6-f54afa0ec34d\") " pod="e2e-models-e2e-worker-w3-7a8fa2/premium-w3-7a8fa2-kserve-6b6c5f6fb8-n5wtq" Oct 30 18:44:09.789595 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:09.789559 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f51fd746-cd85-4608-96a6-f54afa0ec34d-dshm\") pod \"premium-w3-7a8fa2-kserve-6b6c5f6fb8-n5wtq\" (UID: \"f51fd746-cd85-4608-96a6-f54afa0ec34d\") " pod="e2e-models-e2e-worker-w3-7a8fa2/premium-w3-7a8fa2-kserve-6b6c5f6fb8-n5wtq" Oct 30 18:44:09.789595 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:09.789600 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f51fd746-cd85-4608-96a6-f54afa0ec34d-home\") pod \"premium-w3-7a8fa2-kserve-6b6c5f6fb8-n5wtq\" (UID: \"f51fd746-cd85-4608-96a6-f54afa0ec34d\") " pod="e2e-models-e2e-worker-w3-7a8fa2/premium-w3-7a8fa2-kserve-6b6c5f6fb8-n5wtq" Oct 30 18:44:09.789858 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:09.789741 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f51fd746-cd85-4608-96a6-f54afa0ec34d-tls-certs\") pod \"premium-w3-7a8fa2-kserve-6b6c5f6fb8-n5wtq\" (UID: \"f51fd746-cd85-4608-96a6-f54afa0ec34d\") " pod="e2e-models-e2e-worker-w3-7a8fa2/premium-w3-7a8fa2-kserve-6b6c5f6fb8-n5wtq" Oct 30 18:44:09.789858 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:09.789786 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kl582\" (UniqueName: \"kubernetes.io/projected/f51fd746-cd85-4608-96a6-f54afa0ec34d-kube-api-access-kl582\") pod \"premium-w3-7a8fa2-kserve-6b6c5f6fb8-n5wtq\" (UID: \"f51fd746-cd85-4608-96a6-f54afa0ec34d\") " pod="e2e-models-e2e-worker-w3-7a8fa2/premium-w3-7a8fa2-kserve-6b6c5f6fb8-n5wtq" Oct 30 18:44:09.789858 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:09.789831 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f51fd746-cd85-4608-96a6-f54afa0ec34d-tmp-dir\") pod \"premium-w3-7a8fa2-kserve-6b6c5f6fb8-n5wtq\" (UID: \"f51fd746-cd85-4608-96a6-f54afa0ec34d\") " pod="e2e-models-e2e-worker-w3-7a8fa2/premium-w3-7a8fa2-kserve-6b6c5f6fb8-n5wtq" Oct 30 18:44:09.790022 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:09.789869 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f51fd746-cd85-4608-96a6-f54afa0ec34d-model-cache\") pod \"premium-w3-7a8fa2-kserve-6b6c5f6fb8-n5wtq\" (UID: \"f51fd746-cd85-4608-96a6-f54afa0ec34d\") " pod="e2e-models-e2e-worker-w3-7a8fa2/premium-w3-7a8fa2-kserve-6b6c5f6fb8-n5wtq" Oct 30 18:44:09.790022 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:09.789982 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f51fd746-cd85-4608-96a6-f54afa0ec34d-home\") pod \"premium-w3-7a8fa2-kserve-6b6c5f6fb8-n5wtq\" (UID: \"f51fd746-cd85-4608-96a6-f54afa0ec34d\") " pod="e2e-models-e2e-worker-w3-7a8fa2/premium-w3-7a8fa2-kserve-6b6c5f6fb8-n5wtq" Oct 30 18:44:09.790238 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:09.790214 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f51fd746-cd85-4608-96a6-f54afa0ec34d-model-cache\") pod \"premium-w3-7a8fa2-kserve-6b6c5f6fb8-n5wtq\" (UID: \"f51fd746-cd85-4608-96a6-f54afa0ec34d\") " pod="e2e-models-e2e-worker-w3-7a8fa2/premium-w3-7a8fa2-kserve-6b6c5f6fb8-n5wtq" Oct 30 18:44:09.790478 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:09.790459 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f51fd746-cd85-4608-96a6-f54afa0ec34d-tmp-dir\") pod \"premium-w3-7a8fa2-kserve-6b6c5f6fb8-n5wtq\" (UID: \"f51fd746-cd85-4608-96a6-f54afa0ec34d\") " pod="e2e-models-e2e-worker-w3-7a8fa2/premium-w3-7a8fa2-kserve-6b6c5f6fb8-n5wtq" Oct 30 18:44:09.792201 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:09.792175 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f51fd746-cd85-4608-96a6-f54afa0ec34d-dshm\") pod \"premium-w3-7a8fa2-kserve-6b6c5f6fb8-n5wtq\" (UID: \"f51fd746-cd85-4608-96a6-f54afa0ec34d\") " pod="e2e-models-e2e-worker-w3-7a8fa2/premium-w3-7a8fa2-kserve-6b6c5f6fb8-n5wtq" Oct 30 18:44:09.792397 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:09.792381 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f51fd746-cd85-4608-96a6-f54afa0ec34d-tls-certs\") pod \"premium-w3-7a8fa2-kserve-6b6c5f6fb8-n5wtq\" (UID: \"f51fd746-cd85-4608-96a6-f54afa0ec34d\") " pod="e2e-models-e2e-worker-w3-7a8fa2/premium-w3-7a8fa2-kserve-6b6c5f6fb8-n5wtq" Oct 30 18:44:09.831156 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:09.831114 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kl582\" (UniqueName: \"kubernetes.io/projected/f51fd746-cd85-4608-96a6-f54afa0ec34d-kube-api-access-kl582\") pod \"premium-w3-7a8fa2-kserve-6b6c5f6fb8-n5wtq\" (UID: \"f51fd746-cd85-4608-96a6-f54afa0ec34d\") " pod="e2e-models-e2e-worker-w3-7a8fa2/premium-w3-7a8fa2-kserve-6b6c5f6fb8-n5wtq" Oct 30 18:44:09.915185 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:09.915095 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-7a8fa2/premium-w3-7a8fa2-kserve-6b6c5f6fb8-n5wtq" Oct 30 18:44:09.941612 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:09.941584 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:10.198012 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:10.197984 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-7a8fa2/premium-w3-7a8fa2-kserve-6b6c5f6fb8-n5wtq"] Oct 30 18:44:10.200095 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:44:10.200071 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf51fd746_cd85_4608_96a6_f54afa0ec34d.slice/crio-48cb28d2f5bffe3d94dca03a91ab9c37beff32cb318f25b868b86f9d51a5cef2 WatchSource:0}: Error finding container 48cb28d2f5bffe3d94dca03a91ab9c37beff32cb318f25b868b86f9d51a5cef2: Status 404 returned error can't find the container with id 48cb28d2f5bffe3d94dca03a91ab9c37beff32cb318f25b868b86f9d51a5cef2 Oct 30 18:44:10.611447 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:10.611412 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-7a8fa2/premium-w3-7a8fa2-kserve-6b6c5f6fb8-n5wtq" event={"ID":"f51fd746-cd85-4608-96a6-f54afa0ec34d","Type":"ContainerStarted","Data":"7272ada118e39fc2b5c37624c620ca1ffdf53ff85c9374e52e94e678b28b5c98"} Oct 30 18:44:10.611717 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:10.611691 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-7a8fa2/premium-w3-7a8fa2-kserve-6b6c5f6fb8-n5wtq" event={"ID":"f51fd746-cd85-4608-96a6-f54afa0ec34d","Type":"ContainerStarted","Data":"48cb28d2f5bffe3d94dca03a91ab9c37beff32cb318f25b868b86f9d51a5cef2"} Oct 30 18:44:10.643227 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:10.643173 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-7a8fa2/premium-w3-7a8fa2-kserve-6b6c5f6fb8-n5wtq" podStartSLOduration=1.643138482 podStartE2EDuration="1.643138482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:44:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:44:10.640097526 +0000 UTC m=+618.169128664" watchObservedRunningTime="2026-10-30 18:44:10.643138482 +0000 UTC m=+618.172169623" Oct 30 18:44:10.856691 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:10.856656 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:10.915441 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:10.915367 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7a8fa2/premium-w3-7a8fa2-kserve-6b6c5f6fb8-n5wtq" Oct 30 18:44:11.763610 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:11.763561 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:12.628639 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:12.628602 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:14.020936 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:14.020898 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:14.720113 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:14.720082 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:15.402752 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:15.402714 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:16.407604 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:16.407557 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:17.453734 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:17.453698 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:18.228806 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:18.228774 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:19.044179 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:19.044127 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:19.916056 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:19.916027 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-7a8fa2/premium-w3-7a8fa2-kserve-6b6c5f6fb8-n5wtq" Oct 30 18:44:19.928745 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:19.928723 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-7a8fa2/premium-w3-7a8fa2-kserve-6b6c5f6fb8-n5wtq" Oct 30 18:44:20.127835 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:20.127802 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:20.674119 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:20.674089 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7a8fa2/premium-w3-7a8fa2-kserve-6b6c5f6fb8-n5wtq" Oct 30 18:44:21.139339 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:21.139299 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:22.118759 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:22.118721 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:22.554168 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:22.554059 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:23.031012 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:23.030978 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:23.931695 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:44:23.931657 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 18:44:23.976846 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:23.976809 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:24.674000 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:24.673970 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-aa6736/premium-w1-aa6736-kserve-6f986b66f7-nkfrg"] Oct 30 18:44:24.679284 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:24.679261 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-aa6736/premium-w1-aa6736-kserve-6f986b66f7-nkfrg" Oct 30 18:44:24.690653 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:24.690630 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-aa6736\"/\"premium-w1-aa6736-kserve-self-signed-certs\"" Oct 30 18:44:24.690792 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:24.690773 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-aa6736/premium-w1-aa6736-kserve-6f986b66f7-nkfrg"] Oct 30 18:44:24.746812 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:24.746773 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84ca40c5-7033-48c4-bcec-e9ac7c034a4e-tls-certs\") pod \"premium-w1-aa6736-kserve-6f986b66f7-nkfrg\" (UID: \"84ca40c5-7033-48c4-bcec-e9ac7c034a4e\") " pod="e2e-models-e2e-worker-w1-aa6736/premium-w1-aa6736-kserve-6f986b66f7-nkfrg" Oct 30 18:44:24.746983 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:24.746822 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84ca40c5-7033-48c4-bcec-e9ac7c034a4e-home\") pod \"premium-w1-aa6736-kserve-6f986b66f7-nkfrg\" (UID: \"84ca40c5-7033-48c4-bcec-e9ac7c034a4e\") " pod="e2e-models-e2e-worker-w1-aa6736/premium-w1-aa6736-kserve-6f986b66f7-nkfrg" Oct 30 18:44:24.746983 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:24.746960 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84ca40c5-7033-48c4-bcec-e9ac7c034a4e-tmp-dir\") pod \"premium-w1-aa6736-kserve-6f986b66f7-nkfrg\" (UID: \"84ca40c5-7033-48c4-bcec-e9ac7c034a4e\") " pod="e2e-models-e2e-worker-w1-aa6736/premium-w1-aa6736-kserve-6f986b66f7-nkfrg" Oct 30 18:44:24.747068 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:24.747014 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84ca40c5-7033-48c4-bcec-e9ac7c034a4e-dshm\") pod \"premium-w1-aa6736-kserve-6f986b66f7-nkfrg\" (UID: \"84ca40c5-7033-48c4-bcec-e9ac7c034a4e\") " pod="e2e-models-e2e-worker-w1-aa6736/premium-w1-aa6736-kserve-6f986b66f7-nkfrg" Oct 30 18:44:24.747068 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:24.747064 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84ca40c5-7033-48c4-bcec-e9ac7c034a4e-model-cache\") pod \"premium-w1-aa6736-kserve-6f986b66f7-nkfrg\" (UID: \"84ca40c5-7033-48c4-bcec-e9ac7c034a4e\") " pod="e2e-models-e2e-worker-w1-aa6736/premium-w1-aa6736-kserve-6f986b66f7-nkfrg" Oct 30 18:44:24.747197 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:24.747131 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5xlc\" (UniqueName: \"kubernetes.io/projected/84ca40c5-7033-48c4-bcec-e9ac7c034a4e-kube-api-access-k5xlc\") pod \"premium-w1-aa6736-kserve-6f986b66f7-nkfrg\" (UID: \"84ca40c5-7033-48c4-bcec-e9ac7c034a4e\") " pod="e2e-models-e2e-worker-w1-aa6736/premium-w1-aa6736-kserve-6f986b66f7-nkfrg" Oct 30 18:44:24.781535 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:24.781502 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:24.849057 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:24.849023 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84ca40c5-7033-48c4-bcec-e9ac7c034a4e-tls-certs\") pod \"premium-w1-aa6736-kserve-6f986b66f7-nkfrg\" (UID: \"84ca40c5-7033-48c4-bcec-e9ac7c034a4e\") " pod="e2e-models-e2e-worker-w1-aa6736/premium-w1-aa6736-kserve-6f986b66f7-nkfrg" Oct 30 18:44:24.849311 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:24.849066 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84ca40c5-7033-48c4-bcec-e9ac7c034a4e-home\") pod \"premium-w1-aa6736-kserve-6f986b66f7-nkfrg\" (UID: \"84ca40c5-7033-48c4-bcec-e9ac7c034a4e\") " pod="e2e-models-e2e-worker-w1-aa6736/premium-w1-aa6736-kserve-6f986b66f7-nkfrg" Oct 30 18:44:24.849311 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:24.849135 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84ca40c5-7033-48c4-bcec-e9ac7c034a4e-tmp-dir\") pod \"premium-w1-aa6736-kserve-6f986b66f7-nkfrg\" (UID: \"84ca40c5-7033-48c4-bcec-e9ac7c034a4e\") " pod="e2e-models-e2e-worker-w1-aa6736/premium-w1-aa6736-kserve-6f986b66f7-nkfrg" Oct 30 18:44:24.849311 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:24.849222 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84ca40c5-7033-48c4-bcec-e9ac7c034a4e-dshm\") pod \"premium-w1-aa6736-kserve-6f986b66f7-nkfrg\" (UID: \"84ca40c5-7033-48c4-bcec-e9ac7c034a4e\") " pod="e2e-models-e2e-worker-w1-aa6736/premium-w1-aa6736-kserve-6f986b66f7-nkfrg" Oct 30 18:44:24.849311 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:24.849285 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84ca40c5-7033-48c4-bcec-e9ac7c034a4e-model-cache\") pod \"premium-w1-aa6736-kserve-6f986b66f7-nkfrg\" (UID: \"84ca40c5-7033-48c4-bcec-e9ac7c034a4e\") " pod="e2e-models-e2e-worker-w1-aa6736/premium-w1-aa6736-kserve-6f986b66f7-nkfrg" Oct 30 18:44:24.849536 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:24.849320 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5xlc\" (UniqueName: \"kubernetes.io/projected/84ca40c5-7033-48c4-bcec-e9ac7c034a4e-kube-api-access-k5xlc\") pod \"premium-w1-aa6736-kserve-6f986b66f7-nkfrg\" (UID: \"84ca40c5-7033-48c4-bcec-e9ac7c034a4e\") " pod="e2e-models-e2e-worker-w1-aa6736/premium-w1-aa6736-kserve-6f986b66f7-nkfrg" Oct 30 18:44:24.849906 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:24.849877 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84ca40c5-7033-48c4-bcec-e9ac7c034a4e-tmp-dir\") pod \"premium-w1-aa6736-kserve-6f986b66f7-nkfrg\" (UID: \"84ca40c5-7033-48c4-bcec-e9ac7c034a4e\") " pod="e2e-models-e2e-worker-w1-aa6736/premium-w1-aa6736-kserve-6f986b66f7-nkfrg" Oct 30 18:44:24.850189 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:24.850134 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84ca40c5-7033-48c4-bcec-e9ac7c034a4e-home\") pod \"premium-w1-aa6736-kserve-6f986b66f7-nkfrg\" (UID: \"84ca40c5-7033-48c4-bcec-e9ac7c034a4e\") " pod="e2e-models-e2e-worker-w1-aa6736/premium-w1-aa6736-kserve-6f986b66f7-nkfrg" Oct 30 18:44:24.850373 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:24.850359 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84ca40c5-7033-48c4-bcec-e9ac7c034a4e-model-cache\") pod \"premium-w1-aa6736-kserve-6f986b66f7-nkfrg\" (UID: \"84ca40c5-7033-48c4-bcec-e9ac7c034a4e\") " pod="e2e-models-e2e-worker-w1-aa6736/premium-w1-aa6736-kserve-6f986b66f7-nkfrg" Oct 30 18:44:24.852967 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:24.852943 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84ca40c5-7033-48c4-bcec-e9ac7c034a4e-tls-certs\") pod \"premium-w1-aa6736-kserve-6f986b66f7-nkfrg\" (UID: \"84ca40c5-7033-48c4-bcec-e9ac7c034a4e\") " pod="e2e-models-e2e-worker-w1-aa6736/premium-w1-aa6736-kserve-6f986b66f7-nkfrg" Oct 30 18:44:24.853197 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:24.853176 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84ca40c5-7033-48c4-bcec-e9ac7c034a4e-dshm\") pod \"premium-w1-aa6736-kserve-6f986b66f7-nkfrg\" (UID: \"84ca40c5-7033-48c4-bcec-e9ac7c034a4e\") " pod="e2e-models-e2e-worker-w1-aa6736/premium-w1-aa6736-kserve-6f986b66f7-nkfrg" Oct 30 18:44:24.862570 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:24.862550 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5xlc\" (UniqueName: \"kubernetes.io/projected/84ca40c5-7033-48c4-bcec-e9ac7c034a4e-kube-api-access-k5xlc\") pod \"premium-w1-aa6736-kserve-6f986b66f7-nkfrg\" (UID: \"84ca40c5-7033-48c4-bcec-e9ac7c034a4e\") " pod="e2e-models-e2e-worker-w1-aa6736/premium-w1-aa6736-kserve-6f986b66f7-nkfrg" Oct 30 18:44:24.971322 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:24.971223 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:24.994388 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:24.994355 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-aa6736/premium-w1-aa6736-kserve-6f986b66f7-nkfrg" Oct 30 18:44:25.273095 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:25.273072 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-aa6736/premium-w1-aa6736-kserve-6f986b66f7-nkfrg"] Oct 30 18:44:25.275012 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:44:25.274955 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod84ca40c5_7033_48c4_bcec_e9ac7c034a4e.slice/crio-8613f8c5399726db6bda0c14bc3635db951a6a22bddf75bc1f75d9855f9a6577 WatchSource:0}: Error finding container 8613f8c5399726db6bda0c14bc3635db951a6a22bddf75bc1f75d9855f9a6577: Status 404 returned error can't find the container with id 8613f8c5399726db6bda0c14bc3635db951a6a22bddf75bc1f75d9855f9a6577 Oct 30 18:44:25.691676 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:25.691643 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-aa6736/premium-w1-aa6736-kserve-6f986b66f7-nkfrg" event={"ID":"84ca40c5-7033-48c4-bcec-e9ac7c034a4e","Type":"ContainerStarted","Data":"3f5ce52a570bb1a7e8b84a47e71b27cbcb6454e8f194ce65a6e555ae7b5bb2b2"} Oct 30 18:44:25.691676 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:25.691687 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-aa6736/premium-w1-aa6736-kserve-6f986b66f7-nkfrg" event={"ID":"84ca40c5-7033-48c4-bcec-e9ac7c034a4e","Type":"ContainerStarted","Data":"8613f8c5399726db6bda0c14bc3635db951a6a22bddf75bc1f75d9855f9a6577"} Oct 30 18:44:25.727297 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:25.727168 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-aa6736/premium-w1-aa6736-kserve-6f986b66f7-nkfrg" podStartSLOduration=1.7271355179999999 podStartE2EDuration="1.727135518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:44:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:44:25.725662136 +0000 UTC m=+633.254693275" watchObservedRunningTime="2026-10-30 18:44:25.727135518 +0000 UTC m=+633.256166654" Oct 30 18:44:25.975053 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:25.974940 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:25.995188 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:25.995129 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-aa6736/premium-w1-aa6736-kserve-6f986b66f7-nkfrg" Oct 30 18:44:27.046956 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:27.046919 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:27.998523 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:27.998489 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:28.905553 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:28.905523 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:29.896296 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:29.896267 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:30.726118 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:30.726074 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:31.793536 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:31.793494 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:32.620501 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:32.620443 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:33.609383 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:33.609310 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:34.554703 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:34.554664 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:34.994990 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:34.994916 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-aa6736/premium-w1-aa6736-kserve-6f986b66f7-nkfrg" Oct 30 18:44:35.007999 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:35.007976 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-aa6736/premium-w1-aa6736-kserve-6f986b66f7-nkfrg" Oct 30 18:44:35.701467 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:35.701427 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:35.755879 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:35.755845 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-aa6736/premium-w1-aa6736-kserve-6f986b66f7-nkfrg" Oct 30 18:44:37.088579 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:37.088547 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:38.724576 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:38.724541 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:39.762358 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:39.762323 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:40.961694 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:40.961647 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:41.928299 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:41.928255 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:42.864023 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:42.863989 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:44.141720 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:44.141685 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:45.572354 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:45.572316 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:47.230782 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:47.230748 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:48.340769 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:48.340726 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:49.357255 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:49.357217 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:50.324533 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:50.324496 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:50.584417 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:50.584333 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:51.710280 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:51.710246 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:52.626357 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:52.626326 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:53.623036 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:53.622997 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:53.932010 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:44:53.931901 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 18:44:54.649649 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:54.649616 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:55.480437 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:55.480400 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:56.337085 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:56.337045 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:57.210933 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:57.210873 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:58.072486 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:58.072455 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:58.202735 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:58.202705 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:58.830841 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:58.830800 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:59.607183 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:44:59.607128 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:00.186443 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:00.186412 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889765-4wkt5"] Oct 30 18:45:00.191663 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:00.191641 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889765-4wkt5" Oct 30 18:45:00.205334 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:00.205312 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889765-4wkt5"] Oct 30 18:45:00.290122 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:00.290091 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\"" Oct 30 18:45:00.315769 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:00.315717 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pl2pn\" (UniqueName: \"kubernetes.io/projected/8ab372c3-e49d-4039-92d9-1f4a3d526497-kube-api-access-pl2pn\") pod \"maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889765-4wkt5\" (UID: \"8ab372c3-e49d-4039-92d9-1f4a3d526497\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889765-4wkt5" Oct 30 18:45:00.315926 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:00.315806 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ab372c3-e49d-4039-92d9-1f4a3d526497-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889765-4wkt5\" (UID: \"8ab372c3-e49d-4039-92d9-1f4a3d526497\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889765-4wkt5" Oct 30 18:45:00.416600 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:00.416558 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pl2pn\" (UniqueName: \"kubernetes.io/projected/8ab372c3-e49d-4039-92d9-1f4a3d526497-kube-api-access-pl2pn\") pod \"maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889765-4wkt5\" (UID: \"8ab372c3-e49d-4039-92d9-1f4a3d526497\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889765-4wkt5" Oct 30 18:45:00.416778 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:00.416636 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ab372c3-e49d-4039-92d9-1f4a3d526497-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889765-4wkt5\" (UID: \"8ab372c3-e49d-4039-92d9-1f4a3d526497\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889765-4wkt5" Oct 30 18:45:00.417348 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:00.417328 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ab372c3-e49d-4039-92d9-1f4a3d526497-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889765-4wkt5\" (UID: \"8ab372c3-e49d-4039-92d9-1f4a3d526497\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889765-4wkt5" Oct 30 18:45:00.430223 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:00.430199 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pl2pn\" (UniqueName: \"kubernetes.io/projected/8ab372c3-e49d-4039-92d9-1f4a3d526497-kube-api-access-pl2pn\") pod \"maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889765-4wkt5\" (UID: \"8ab372c3-e49d-4039-92d9-1f4a3d526497\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889765-4wkt5" Oct 30 18:45:00.506193 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:00.506099 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889765-4wkt5" Oct 30 18:45:00.748792 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:00.748752 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:00.794866 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:00.794840 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889765-4wkt5"] Oct 30 18:45:00.796241 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:45:00.796216 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ab372c3_e49d_4039_92d9_1f4a3d526497.slice/crio-0c8c3657486656889632c4835a42a18f4b5ee36b0810ddcf5c8d8f54d1f80dac WatchSource:0}: Error finding container 0c8c3657486656889632c4835a42a18f4b5ee36b0810ddcf5c8d8f54d1f80dac: Status 404 returned error can't find the container with id 0c8c3657486656889632c4835a42a18f4b5ee36b0810ddcf5c8d8f54d1f80dac Oct 30 18:45:00.798376 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:00.798357 6960 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 18:45:00.877315 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:00.877285 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889765-4wkt5" event={"ID":"8ab372c3-e49d-4039-92d9-1f4a3d526497","Type":"ContainerStarted","Data":"0c8c3657486656889632c4835a42a18f4b5ee36b0810ddcf5c8d8f54d1f80dac"} Oct 30 18:45:01.723453 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:01.723409 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:02.848103 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:02.848065 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:04.046049 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:04.046009 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:04.902868 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:04.902834 6960 generic.go:356] "Generic (PLEG): container finished" podID="8ab372c3-e49d-4039-92d9-1f4a3d526497" containerID="533b16b70966bd59350aa243bc42fdf8806551eab94ebd00030ff34e4287f1bb" exitCode=60 Oct 30 18:45:04.903038 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:04.902925 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889765-4wkt5" event={"ID":"8ab372c3-e49d-4039-92d9-1f4a3d526497","Type":"ContainerDied","Data":"533b16b70966bd59350aa243bc42fdf8806551eab94ebd00030ff34e4287f1bb"} Oct 30 18:45:04.903219 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:04.903203 6960 scope.go:122] "RemoveContainer" containerID="533b16b70966bd59350aa243bc42fdf8806551eab94ebd00030ff34e4287f1bb" Oct 30 18:45:04.940670 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:04.940649 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:05.909955 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:05.909867 6960 generic.go:356] "Generic (PLEG): container finished" podID="8ab372c3-e49d-4039-92d9-1f4a3d526497" containerID="3c52e3f5bcc36bbc491045d18fa8526d0a7d144af99152f0ab43372b3b5388a1" exitCode=60 Oct 30 18:45:05.910418 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:05.909954 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889765-4wkt5" event={"ID":"8ab372c3-e49d-4039-92d9-1f4a3d526497","Type":"ContainerDied","Data":"3c52e3f5bcc36bbc491045d18fa8526d0a7d144af99152f0ab43372b3b5388a1"} Oct 30 18:45:05.910418 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:05.910014 6960 scope.go:122] "RemoveContainer" containerID="533b16b70966bd59350aa243bc42fdf8806551eab94ebd00030ff34e4287f1bb" Oct 30 18:45:05.910418 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:05.910378 6960 scope.go:122] "RemoveContainer" containerID="3c52e3f5bcc36bbc491045d18fa8526d0a7d144af99152f0ab43372b3b5388a1" Oct 30 18:45:05.910689 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:45:05.910664 6960 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889765-4wkt5_odh-ai-gateway-infra(8ab372c3-e49d-4039-92d9-1f4a3d526497)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889765-4wkt5" podUID="8ab372c3-e49d-4039-92d9-1f4a3d526497" Oct 30 18:45:05.946967 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:05.946922 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:06.918123 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:06.918098 6960 scope.go:122] "RemoveContainer" containerID="3c52e3f5bcc36bbc491045d18fa8526d0a7d144af99152f0ab43372b3b5388a1" Oct 30 18:45:06.918651 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:45:06.918326 6960 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889765-4wkt5_odh-ai-gateway-infra(8ab372c3-e49d-4039-92d9-1f4a3d526497)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889765-4wkt5" podUID="8ab372c3-e49d-4039-92d9-1f4a3d526497" Oct 30 18:45:06.987486 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:06.987456 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:07.892418 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:07.892384 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:08.862916 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:08.862876 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:09.268348 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:09.268275 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ace9e50-45s55"] Oct 30 18:45:09.272424 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:09.272403 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ace9e50-45s55" Oct 30 18:45:09.275361 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:09.275333 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:45:09.281419 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:09.281397 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ace9e50-45s55"] Oct 30 18:45:09.306551 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:09.306514 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pw6l\" (UniqueName: \"kubernetes.io/projected/6c6b8bf7-c56e-4497-a01c-cdaae9975f44-kube-api-access-6pw6l\") pod \"maas-api-revoke-sub-7ace9e50-45s55\" (UID: \"6c6b8bf7-c56e-4497-a01c-cdaae9975f44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ace9e50-45s55" Oct 30 18:45:09.306709 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:09.306689 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c6b8bf7-c56e-4497-a01c-cdaae9975f44-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ace9e50-45s55\" (UID: \"6c6b8bf7-c56e-4497-a01c-cdaae9975f44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ace9e50-45s55" Oct 30 18:45:09.306778 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:09.306726 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c6b8bf7-c56e-4497-a01c-cdaae9975f44-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ace9e50-45s55\" (UID: \"6c6b8bf7-c56e-4497-a01c-cdaae9975f44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ace9e50-45s55" Oct 30 18:45:09.408088 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:09.408056 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c6b8bf7-c56e-4497-a01c-cdaae9975f44-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ace9e50-45s55\" (UID: \"6c6b8bf7-c56e-4497-a01c-cdaae9975f44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ace9e50-45s55" Oct 30 18:45:09.408311 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:09.408099 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c6b8bf7-c56e-4497-a01c-cdaae9975f44-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ace9e50-45s55\" (UID: \"6c6b8bf7-c56e-4497-a01c-cdaae9975f44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ace9e50-45s55" Oct 30 18:45:09.408311 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:09.408179 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pw6l\" (UniqueName: \"kubernetes.io/projected/6c6b8bf7-c56e-4497-a01c-cdaae9975f44-kube-api-access-6pw6l\") pod \"maas-api-revoke-sub-7ace9e50-45s55\" (UID: \"6c6b8bf7-c56e-4497-a01c-cdaae9975f44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ace9e50-45s55" Oct 30 18:45:09.408839 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:09.408807 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c6b8bf7-c56e-4497-a01c-cdaae9975f44-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ace9e50-45s55\" (UID: \"6c6b8bf7-c56e-4497-a01c-cdaae9975f44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ace9e50-45s55" Oct 30 18:45:09.410734 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:09.410710 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c6b8bf7-c56e-4497-a01c-cdaae9975f44-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ace9e50-45s55\" (UID: \"6c6b8bf7-c56e-4497-a01c-cdaae9975f44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ace9e50-45s55" Oct 30 18:45:09.420489 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:09.420470 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pw6l\" (UniqueName: \"kubernetes.io/projected/6c6b8bf7-c56e-4497-a01c-cdaae9975f44-kube-api-access-6pw6l\") pod \"maas-api-revoke-sub-7ace9e50-45s55\" (UID: \"6c6b8bf7-c56e-4497-a01c-cdaae9975f44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ace9e50-45s55" Oct 30 18:45:09.588380 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:09.588349 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ace9e50-45s55" Oct 30 18:45:09.705315 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:09.705284 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:09.863986 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:09.863965 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ace9e50-45s55"] Oct 30 18:45:09.865526 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:45:09.865499 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c6b8bf7_c56e_4497_a01c_cdaae9975f44.slice/crio-6d733e8b4cbf7b9a2f088bd63286d56eb088b66f827b2aa786b81ce48d32909f WatchSource:0}: Error finding container 6d733e8b4cbf7b9a2f088bd63286d56eb088b66f827b2aa786b81ce48d32909f: Status 404 returned error can't find the container with id 6d733e8b4cbf7b9a2f088bd63286d56eb088b66f827b2aa786b81ce48d32909f Oct 30 18:45:09.935124 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:09.935100 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ace9e50-45s55" event={"ID":"6c6b8bf7-c56e-4497-a01c-cdaae9975f44","Type":"ContainerStarted","Data":"6d733e8b4cbf7b9a2f088bd63286d56eb088b66f827b2aa786b81ce48d32909f"} Oct 30 18:45:10.736462 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:10.736431 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:10.942431 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:10.942337 6960 generic.go:356] "Generic (PLEG): container finished" podID="6c6b8bf7-c56e-4497-a01c-cdaae9975f44" containerID="820c9ee17cf20f5cd7bea3349236ec71c9747ebc77683e654ad364ffb2b76ff8" exitCode=0 Oct 30 18:45:10.942431 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:10.942370 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ace9e50-45s55" event={"ID":"6c6b8bf7-c56e-4497-a01c-cdaae9975f44","Type":"ContainerDied","Data":"820c9ee17cf20f5cd7bea3349236ec71c9747ebc77683e654ad364ffb2b76ff8"} Oct 30 18:45:11.756483 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:11.756449 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:12.086193 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:12.086132 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ace9e50-45s55" Oct 30 18:45:12.137786 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:12.137757 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c6b8bf7-c56e-4497-a01c-cdaae9975f44-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c6b8bf7-c56e-4497-a01c-cdaae9975f44-maas-api-openshift-ca\") pod \"6c6b8bf7-c56e-4497-a01c-cdaae9975f44\" (UID: \"6c6b8bf7-c56e-4497-a01c-cdaae9975f44\") " Oct 30 18:45:12.137904 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:12.137830 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c6b8bf7-c56e-4497-a01c-cdaae9975f44-kube-api-access-6pw6l\" (UniqueName: \"kubernetes.io/projected/6c6b8bf7-c56e-4497-a01c-cdaae9975f44-kube-api-access-6pw6l\") pod \"6c6b8bf7-c56e-4497-a01c-cdaae9975f44\" (UID: \"6c6b8bf7-c56e-4497-a01c-cdaae9975f44\") " Oct 30 18:45:12.137904 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:12.137876 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c6b8bf7-c56e-4497-a01c-cdaae9975f44-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c6b8bf7-c56e-4497-a01c-cdaae9975f44-maas-api-xks-ca\") pod \"6c6b8bf7-c56e-4497-a01c-cdaae9975f44\" (UID: \"6c6b8bf7-c56e-4497-a01c-cdaae9975f44\") " Oct 30 18:45:12.138226 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:12.138193 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c6b8bf7-c56e-4497-a01c-cdaae9975f44-maas-api-openshift-ca" pod "6c6b8bf7-c56e-4497-a01c-cdaae9975f44" (UID: "6c6b8bf7-c56e-4497-a01c-cdaae9975f44"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:45:12.138372 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:12.138293 6960 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c6b8bf7-c56e-4497-a01c-cdaae9975f44-maas-api-openshift-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:45:12.140078 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:12.140051 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c6b8bf7-c56e-4497-a01c-cdaae9975f44-kube-api-access-6pw6l" pod "6c6b8bf7-c56e-4497-a01c-cdaae9975f44" (UID: "6c6b8bf7-c56e-4497-a01c-cdaae9975f44"). InnerVolumeSpecName "kube-api-access-6pw6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:45:12.140204 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:12.140083 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c6b8bf7-c56e-4497-a01c-cdaae9975f44-maas-api-xks-ca" pod "6c6b8bf7-c56e-4497-a01c-cdaae9975f44" (UID: "6c6b8bf7-c56e-4497-a01c-cdaae9975f44"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:45:12.239312 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:12.239281 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pw6l\" (UniqueName: \"kubernetes.io/projected/6c6b8bf7-c56e-4497-a01c-cdaae9975f44-kube-api-access-6pw6l\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:45:12.239312 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:12.239307 6960 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c6b8bf7-c56e-4497-a01c-cdaae9975f44-maas-api-xks-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:45:12.725162 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:12.725120 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:12.954332 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:12.954256 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ace9e50-45s55" Oct 30 18:45:12.954473 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:12.954256 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ace9e50-45s55" event={"ID":"6c6b8bf7-c56e-4497-a01c-cdaae9975f44","Type":"ContainerDied","Data":"6d733e8b4cbf7b9a2f088bd63286d56eb088b66f827b2aa786b81ce48d32909f"} Oct 30 18:45:12.954473 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:12.954372 6960 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d733e8b4cbf7b9a2f088bd63286d56eb088b66f827b2aa786b81ce48d32909f" Oct 30 18:45:13.140273 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:13.140229 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:13.621318 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:13.621285 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:14.615487 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:14.615454 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:15.522681 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:15.522640 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:16.317250 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:16.317216 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:17.126191 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:17.126133 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:17.927218 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:17.927178 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:18.585641 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:18.585604 6960 scope.go:122] "RemoveContainer" containerID="3c52e3f5bcc36bbc491045d18fa8526d0a7d144af99152f0ab43372b3b5388a1" Oct 30 18:45:18.714618 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:18.714530 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:18.987684 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:18.987650 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889765-4wkt5" event={"ID":"8ab372c3-e49d-4039-92d9-1f4a3d526497","Type":"ContainerStarted","Data":"7c38673c4323cd7118533c72662769d18f9e7bb43546916869e767072a438f47"} Oct 30 18:45:19.009832 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:19.009786 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889765-4wkt5" podStartSLOduration=16.16937961 podStartE2EDuration="19.00977285s" totalImagesPullingTime="2.84039324s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:19.007757716 +0000 UTC m=+686.536788853" watchObservedRunningTime="2026-10-30 18:45:19.00977285 +0000 UTC m=+686.538803987" Oct 30 18:45:19.708120 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:19.708087 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:19.994881 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:19.994793 6960 generic.go:356] "Generic (PLEG): container finished" podID="8ab372c3-e49d-4039-92d9-1f4a3d526497" containerID="7c38673c4323cd7118533c72662769d18f9e7bb43546916869e767072a438f47" exitCode=60 Oct 30 18:45:19.995301 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:19.994873 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889765-4wkt5" event={"ID":"8ab372c3-e49d-4039-92d9-1f4a3d526497","Type":"ContainerDied","Data":"7c38673c4323cd7118533c72662769d18f9e7bb43546916869e767072a438f47"} Oct 30 18:45:19.995301 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:19.994929 6960 scope.go:122] "RemoveContainer" containerID="3c52e3f5bcc36bbc491045d18fa8526d0a7d144af99152f0ab43372b3b5388a1" Oct 30 18:45:19.995301 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:19.995263 6960 scope.go:122] "RemoveContainer" containerID="7c38673c4323cd7118533c72662769d18f9e7bb43546916869e767072a438f47" Oct 30 18:45:19.995558 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:45:19.995535 6960 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889765-4wkt5_odh-ai-gateway-infra(8ab372c3-e49d-4039-92d9-1f4a3d526497)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889765-4wkt5" podUID="8ab372c3-e49d-4039-92d9-1f4a3d526497" Oct 30 18:45:20.023362 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:20.023336 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889765-4wkt5"] Oct 30 18:45:21.090871 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:21.090841 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:21.134575 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:21.134551 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889765-4wkt5" Oct 30 18:45:21.231239 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:21.231209 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ab372c3-e49d-4039-92d9-1f4a3d526497-kube-api-access-pl2pn\" (UniqueName: \"kubernetes.io/projected/8ab372c3-e49d-4039-92d9-1f4a3d526497-kube-api-access-pl2pn\") pod \"8ab372c3-e49d-4039-92d9-1f4a3d526497\" (UID: \"8ab372c3-e49d-4039-92d9-1f4a3d526497\") " Oct 30 18:45:21.231464 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:21.231446 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ab372c3-e49d-4039-92d9-1f4a3d526497-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ab372c3-e49d-4039-92d9-1f4a3d526497-maas-api-service-ca\") pod \"8ab372c3-e49d-4039-92d9-1f4a3d526497\" (UID: \"8ab372c3-e49d-4039-92d9-1f4a3d526497\") " Oct 30 18:45:21.231757 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:21.231737 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ab372c3-e49d-4039-92d9-1f4a3d526497-maas-api-service-ca" pod "8ab372c3-e49d-4039-92d9-1f4a3d526497" (UID: "8ab372c3-e49d-4039-92d9-1f4a3d526497"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:45:21.233332 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:21.233312 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ab372c3-e49d-4039-92d9-1f4a3d526497-kube-api-access-pl2pn" pod "8ab372c3-e49d-4039-92d9-1f4a3d526497" (UID: "8ab372c3-e49d-4039-92d9-1f4a3d526497"). InnerVolumeSpecName "kube-api-access-pl2pn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:45:21.332664 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:21.332633 6960 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ab372c3-e49d-4039-92d9-1f4a3d526497-maas-api-service-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:45:21.332664 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:21.332660 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pl2pn\" (UniqueName: \"kubernetes.io/projected/8ab372c3-e49d-4039-92d9-1f4a3d526497-kube-api-access-pl2pn\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:45:22.007849 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:22.007816 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889765-4wkt5" event={"ID":"8ab372c3-e49d-4039-92d9-1f4a3d526497","Type":"ContainerDied","Data":"0c8c3657486656889632c4835a42a18f4b5ee36b0810ddcf5c8d8f54d1f80dac"} Oct 30 18:45:22.008038 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:22.007856 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889765-4wkt5" Oct 30 18:45:22.008038 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:22.007898 6960 scope.go:122] "RemoveContainer" containerID="7c38673c4323cd7118533c72662769d18f9e7bb43546916869e767072a438f47" Oct 30 18:45:22.029684 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:22.029658 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889765-4wkt5"] Oct 30 18:45:22.039124 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:22.039102 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889765-4wkt5"] Oct 30 18:45:22.112960 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:22.112925 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:22.591907 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:22.591880 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ab372c3-e49d-4039-92d9-1f4a3d526497" path="/var/lib/kubelet/pods/8ab372c3-e49d-4039-92d9-1f4a3d526497/volumes" Oct 30 18:45:23.022555 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:23.022462 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:23.932596 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:45:23.932565 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 18:45:24.004423 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:24.004388 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:24.809589 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:24.809558 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:25.762112 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:25.762069 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:26.711335 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:26.711303 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:26.959138 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:26.959108 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-m5wdn"] Oct 30 18:45:26.959615 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:26.959600 6960 cpu_manager.go:414] "containerMap: removing container" podUID="8ab372c3-e49d-4039-92d9-1f4a3d526497" containerName="cleanup" Oct 30 18:45:26.959615 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:26.959615 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ab372c3-e49d-4039-92d9-1f4a3d526497" containerName="cleanup" Oct 30 18:45:26.959703 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:26.959632 6960 cpu_manager.go:414] "containerMap: removing container" podUID="8ab372c3-e49d-4039-92d9-1f4a3d526497" containerName="cleanup" Oct 30 18:45:26.959703 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:26.959636 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ab372c3-e49d-4039-92d9-1f4a3d526497" containerName="cleanup" Oct 30 18:45:26.959703 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:26.959653 6960 cpu_manager.go:414] "containerMap: removing container" podUID="8ab372c3-e49d-4039-92d9-1f4a3d526497" containerName="cleanup" Oct 30 18:45:26.959703 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:26.959658 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ab372c3-e49d-4039-92d9-1f4a3d526497" containerName="cleanup" Oct 30 18:45:26.959703 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:26.959667 6960 cpu_manager.go:414] "containerMap: removing container" podUID="6c6b8bf7-c56e-4497-a01c-cdaae9975f44" containerName="revoke-keys" Oct 30 18:45:26.959703 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:26.959672 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c6b8bf7-c56e-4497-a01c-cdaae9975f44" containerName="revoke-keys" Oct 30 18:45:26.959870 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:26.959752 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ab372c3-e49d-4039-92d9-1f4a3d526497" containerName="cleanup" Oct 30 18:45:26.959870 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:26.959763 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ab372c3-e49d-4039-92d9-1f4a3d526497" containerName="cleanup" Oct 30 18:45:26.959870 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:26.959770 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c6b8bf7-c56e-4497-a01c-cdaae9975f44" containerName="revoke-keys" Oct 30 18:45:26.964787 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:26.964728 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-m5wdn" Oct 30 18:45:26.970886 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:26.970867 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-7a8fa2-openshift-default-dockercfg-dddfr\"" Oct 30 18:45:26.977647 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:26.977621 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-m5wdn"] Oct 30 18:45:27.094653 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:27.094615 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6178626a-4928-40ee-b26a-362f39ee1965-workload-socket\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-m5wdn\" (UID: \"6178626a-4928-40ee-b26a-362f39ee1965\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-m5wdn" Oct 30 18:45:27.094847 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:27.094749 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6178626a-4928-40ee-b26a-362f39ee1965-credential-socket\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-m5wdn\" (UID: \"6178626a-4928-40ee-b26a-362f39ee1965\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-m5wdn" Oct 30 18:45:27.094847 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:27.094785 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6178626a-4928-40ee-b26a-362f39ee1965-istio-data\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-m5wdn\" (UID: \"6178626a-4928-40ee-b26a-362f39ee1965\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-m5wdn" Oct 30 18:45:27.094986 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:27.094840 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6178626a-4928-40ee-b26a-362f39ee1965-istio-podinfo\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-m5wdn\" (UID: \"6178626a-4928-40ee-b26a-362f39ee1965\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-m5wdn" Oct 30 18:45:27.094986 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:27.094914 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6178626a-4928-40ee-b26a-362f39ee1965-istiod-ca-cert\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-m5wdn\" (UID: \"6178626a-4928-40ee-b26a-362f39ee1965\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-m5wdn" Oct 30 18:45:27.095092 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:27.095024 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nw4bw\" (UniqueName: \"kubernetes.io/projected/6178626a-4928-40ee-b26a-362f39ee1965-kube-api-access-nw4bw\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-m5wdn\" (UID: \"6178626a-4928-40ee-b26a-362f39ee1965\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-m5wdn" Oct 30 18:45:27.095092 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:27.095057 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6178626a-4928-40ee-b26a-362f39ee1965-workload-certs\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-m5wdn\" (UID: \"6178626a-4928-40ee-b26a-362f39ee1965\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-m5wdn" Oct 30 18:45:27.095232 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:27.095206 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6178626a-4928-40ee-b26a-362f39ee1965-istio-envoy\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-m5wdn\" (UID: \"6178626a-4928-40ee-b26a-362f39ee1965\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-m5wdn" Oct 30 18:45:27.095276 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:27.095231 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6178626a-4928-40ee-b26a-362f39ee1965-istio-token\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-m5wdn\" (UID: \"6178626a-4928-40ee-b26a-362f39ee1965\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-m5wdn" Oct 30 18:45:27.195776 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:27.195741 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6178626a-4928-40ee-b26a-362f39ee1965-workload-socket\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-m5wdn\" (UID: \"6178626a-4928-40ee-b26a-362f39ee1965\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-m5wdn" Oct 30 18:45:27.195931 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:27.195801 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6178626a-4928-40ee-b26a-362f39ee1965-credential-socket\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-m5wdn\" (UID: \"6178626a-4928-40ee-b26a-362f39ee1965\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-m5wdn" Oct 30 18:45:27.195931 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:27.195823 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6178626a-4928-40ee-b26a-362f39ee1965-istio-data\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-m5wdn\" (UID: \"6178626a-4928-40ee-b26a-362f39ee1965\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-m5wdn" Oct 30 18:45:27.195931 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:27.195837 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6178626a-4928-40ee-b26a-362f39ee1965-istio-podinfo\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-m5wdn\" (UID: \"6178626a-4928-40ee-b26a-362f39ee1965\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-m5wdn" Oct 30 18:45:27.195931 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:27.195855 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6178626a-4928-40ee-b26a-362f39ee1965-istiod-ca-cert\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-m5wdn\" (UID: \"6178626a-4928-40ee-b26a-362f39ee1965\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-m5wdn" Oct 30 18:45:27.195931 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:27.195906 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw4bw\" (UniqueName: \"kubernetes.io/projected/6178626a-4928-40ee-b26a-362f39ee1965-kube-api-access-nw4bw\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-m5wdn\" (UID: \"6178626a-4928-40ee-b26a-362f39ee1965\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-m5wdn" Oct 30 18:45:27.196235 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:27.195933 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6178626a-4928-40ee-b26a-362f39ee1965-workload-certs\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-m5wdn\" (UID: \"6178626a-4928-40ee-b26a-362f39ee1965\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-m5wdn" Oct 30 18:45:27.196235 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:27.195977 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6178626a-4928-40ee-b26a-362f39ee1965-istio-envoy\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-m5wdn\" (UID: \"6178626a-4928-40ee-b26a-362f39ee1965\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-m5wdn" Oct 30 18:45:27.196235 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:27.196003 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6178626a-4928-40ee-b26a-362f39ee1965-istio-token\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-m5wdn\" (UID: \"6178626a-4928-40ee-b26a-362f39ee1965\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-m5wdn" Oct 30 18:45:27.196235 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:27.196213 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6178626a-4928-40ee-b26a-362f39ee1965-workload-socket\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-m5wdn\" (UID: \"6178626a-4928-40ee-b26a-362f39ee1965\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-m5wdn" Oct 30 18:45:27.196235 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:27.196231 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6178626a-4928-40ee-b26a-362f39ee1965-credential-socket\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-m5wdn\" (UID: \"6178626a-4928-40ee-b26a-362f39ee1965\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-m5wdn" Oct 30 18:45:27.196486 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:27.196299 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6178626a-4928-40ee-b26a-362f39ee1965-istio-data\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-m5wdn\" (UID: \"6178626a-4928-40ee-b26a-362f39ee1965\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-m5wdn" Oct 30 18:45:27.196486 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:27.196341 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6178626a-4928-40ee-b26a-362f39ee1965-workload-certs\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-m5wdn\" (UID: \"6178626a-4928-40ee-b26a-362f39ee1965\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-m5wdn" Oct 30 18:45:27.196713 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:27.196689 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6178626a-4928-40ee-b26a-362f39ee1965-istiod-ca-cert\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-m5wdn\" (UID: \"6178626a-4928-40ee-b26a-362f39ee1965\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-m5wdn" Oct 30 18:45:27.198431 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:27.198413 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6178626a-4928-40ee-b26a-362f39ee1965-istio-envoy\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-m5wdn\" (UID: \"6178626a-4928-40ee-b26a-362f39ee1965\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-m5wdn" Oct 30 18:45:27.198592 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:27.198574 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6178626a-4928-40ee-b26a-362f39ee1965-istio-podinfo\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-m5wdn\" (UID: \"6178626a-4928-40ee-b26a-362f39ee1965\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-m5wdn" Oct 30 18:45:27.210380 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:27.210362 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw4bw\" (UniqueName: \"kubernetes.io/projected/6178626a-4928-40ee-b26a-362f39ee1965-kube-api-access-nw4bw\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-m5wdn\" (UID: \"6178626a-4928-40ee-b26a-362f39ee1965\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-m5wdn" Oct 30 18:45:27.210479 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:27.210420 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6178626a-4928-40ee-b26a-362f39ee1965-istio-token\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-m5wdn\" (UID: \"6178626a-4928-40ee-b26a-362f39ee1965\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-m5wdn" Oct 30 18:45:27.279419 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:27.279335 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-m5wdn" Oct 30 18:45:27.314483 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:27.314455 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:27.427972 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:27.427940 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:27.600478 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:27.600447 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-m5wdn"] Oct 30 18:45:27.605540 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:45:27.605512 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6178626a_4928_40ee_b26a_362f39ee1965.slice/crio-89146352917a2fe0095a12dfc90950723a2a375d46b556ed6a8afeae57f749f4 WatchSource:0}: Error finding container 89146352917a2fe0095a12dfc90950723a2a375d46b556ed6a8afeae57f749f4: Status 404 returned error can't find the container with id 89146352917a2fe0095a12dfc90950723a2a375d46b556ed6a8afeae57f749f4 Oct 30 18:45:27.608196 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:27.608130 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:45:27.608285 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:27.608245 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:45:27.608336 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:27.608286 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:45:27.750030 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:27.749991 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:27.924276 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:27.924182 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:28.046242 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:28.046206 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-m5wdn" event={"ID":"6178626a-4928-40ee-b26a-362f39ee1965","Type":"ContainerStarted","Data":"163b6435eaaf59fbcfac836f9281149af66925cf50176ad67369ad5140812263"} Oct 30 18:45:28.046779 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:28.046273 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-m5wdn" event={"ID":"6178626a-4928-40ee-b26a-362f39ee1965","Type":"ContainerStarted","Data":"89146352917a2fe0095a12dfc90950723a2a375d46b556ed6a8afeae57f749f4"} Oct 30 18:45:28.073120 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:28.073069 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-m5wdn" podStartSLOduration=2.073051442 podStartE2EDuration="2.073051442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:45:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:28.069588135 +0000 UTC m=+695.598619272" watchObservedRunningTime="2026-10-30 18:45:28.073051442 +0000 UTC m=+695.602082580" Oct 30 18:45:28.282799 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:28.282282 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-m5wdn" Oct 30 18:45:28.282799 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:28.282416 6960 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-m5wdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Oct 30 18:45:28.282799 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:28.282463 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-m5wdn" podUID="6178626a-4928-40ee-b26a-362f39ee1965" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Oct 30 18:45:29.239682 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:29.239577 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:29.283170 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:29.282526 6960 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-m5wdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Oct 30 18:45:29.283170 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:29.282584 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-m5wdn" podUID="6178626a-4928-40ee-b26a-362f39ee1965" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Oct 30 18:45:29.576169 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:29.568950 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ng5vb"] Oct 30 18:45:29.600169 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:29.586909 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ab372c3-e49d-4039-92d9-1f4a3d526497" containerName="cleanup" Oct 30 18:45:29.610439 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:29.609328 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ng5vb"] Oct 30 18:45:29.610439 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:29.609716 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ng5vb" Oct 30 18:45:29.623673 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:29.622327 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-da70a5-openshift-default-dockercfg-vx9wq\"" Oct 30 18:45:29.737597 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:29.737563 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1ef57ac-d33d-4998-b22b-87b2aed6ba8f-workload-certs\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ng5vb\" (UID: \"f1ef57ac-d33d-4998-b22b-87b2aed6ba8f\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ng5vb" Oct 30 18:45:29.737899 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:29.737878 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1ef57ac-d33d-4998-b22b-87b2aed6ba8f-workload-socket\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ng5vb\" (UID: \"f1ef57ac-d33d-4998-b22b-87b2aed6ba8f\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ng5vb" Oct 30 18:45:29.738100 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:29.738072 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1ef57ac-d33d-4998-b22b-87b2aed6ba8f-istio-envoy\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ng5vb\" (UID: \"f1ef57ac-d33d-4998-b22b-87b2aed6ba8f\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ng5vb" Oct 30 18:45:29.738303 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:29.738285 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb686\" (UniqueName: \"kubernetes.io/projected/f1ef57ac-d33d-4998-b22b-87b2aed6ba8f-kube-api-access-zb686\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ng5vb\" (UID: \"f1ef57ac-d33d-4998-b22b-87b2aed6ba8f\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ng5vb" Oct 30 18:45:29.738444 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:29.738426 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1ef57ac-d33d-4998-b22b-87b2aed6ba8f-credential-socket\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ng5vb\" (UID: \"f1ef57ac-d33d-4998-b22b-87b2aed6ba8f\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ng5vb" Oct 30 18:45:29.738648 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:29.738629 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1ef57ac-d33d-4998-b22b-87b2aed6ba8f-istio-podinfo\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ng5vb\" (UID: \"f1ef57ac-d33d-4998-b22b-87b2aed6ba8f\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ng5vb" Oct 30 18:45:29.738836 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:29.738804 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1ef57ac-d33d-4998-b22b-87b2aed6ba8f-istio-token\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ng5vb\" (UID: \"f1ef57ac-d33d-4998-b22b-87b2aed6ba8f\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ng5vb" Oct 30 18:45:29.739050 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:29.739031 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1ef57ac-d33d-4998-b22b-87b2aed6ba8f-istiod-ca-cert\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ng5vb\" (UID: \"f1ef57ac-d33d-4998-b22b-87b2aed6ba8f\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ng5vb" Oct 30 18:45:29.739307 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:29.739290 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1ef57ac-d33d-4998-b22b-87b2aed6ba8f-istio-data\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ng5vb\" (UID: \"f1ef57ac-d33d-4998-b22b-87b2aed6ba8f\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ng5vb" Oct 30 18:45:29.842464 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:29.840842 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1ef57ac-d33d-4998-b22b-87b2aed6ba8f-workload-certs\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ng5vb\" (UID: \"f1ef57ac-d33d-4998-b22b-87b2aed6ba8f\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ng5vb" Oct 30 18:45:29.842464 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:29.840897 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1ef57ac-d33d-4998-b22b-87b2aed6ba8f-workload-socket\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ng5vb\" (UID: \"f1ef57ac-d33d-4998-b22b-87b2aed6ba8f\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ng5vb" Oct 30 18:45:29.842464 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:29.840943 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1ef57ac-d33d-4998-b22b-87b2aed6ba8f-istio-envoy\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ng5vb\" (UID: \"f1ef57ac-d33d-4998-b22b-87b2aed6ba8f\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ng5vb" Oct 30 18:45:29.842464 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:29.840992 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb686\" (UniqueName: \"kubernetes.io/projected/f1ef57ac-d33d-4998-b22b-87b2aed6ba8f-kube-api-access-zb686\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ng5vb\" (UID: \"f1ef57ac-d33d-4998-b22b-87b2aed6ba8f\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ng5vb" Oct 30 18:45:29.842464 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:29.841021 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1ef57ac-d33d-4998-b22b-87b2aed6ba8f-credential-socket\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ng5vb\" (UID: \"f1ef57ac-d33d-4998-b22b-87b2aed6ba8f\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ng5vb" Oct 30 18:45:29.842464 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:29.841065 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1ef57ac-d33d-4998-b22b-87b2aed6ba8f-istio-podinfo\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ng5vb\" (UID: \"f1ef57ac-d33d-4998-b22b-87b2aed6ba8f\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ng5vb" Oct 30 18:45:29.842464 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:29.841110 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1ef57ac-d33d-4998-b22b-87b2aed6ba8f-istio-token\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ng5vb\" (UID: \"f1ef57ac-d33d-4998-b22b-87b2aed6ba8f\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ng5vb" Oct 30 18:45:29.842464 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:29.841199 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1ef57ac-d33d-4998-b22b-87b2aed6ba8f-istiod-ca-cert\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ng5vb\" (UID: \"f1ef57ac-d33d-4998-b22b-87b2aed6ba8f\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ng5vb" Oct 30 18:45:29.842464 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:29.841277 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1ef57ac-d33d-4998-b22b-87b2aed6ba8f-istio-data\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ng5vb\" (UID: \"f1ef57ac-d33d-4998-b22b-87b2aed6ba8f\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ng5vb" Oct 30 18:45:29.842464 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:29.841933 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1ef57ac-d33d-4998-b22b-87b2aed6ba8f-credential-socket\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ng5vb\" (UID: \"f1ef57ac-d33d-4998-b22b-87b2aed6ba8f\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ng5vb" Oct 30 18:45:29.842464 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:29.842067 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1ef57ac-d33d-4998-b22b-87b2aed6ba8f-workload-socket\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ng5vb\" (UID: \"f1ef57ac-d33d-4998-b22b-87b2aed6ba8f\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ng5vb" Oct 30 18:45:29.842464 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:29.842348 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1ef57ac-d33d-4998-b22b-87b2aed6ba8f-istio-data\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ng5vb\" (UID: \"f1ef57ac-d33d-4998-b22b-87b2aed6ba8f\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ng5vb" Oct 30 18:45:29.843301 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:29.842987 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1ef57ac-d33d-4998-b22b-87b2aed6ba8f-istiod-ca-cert\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ng5vb\" (UID: \"f1ef57ac-d33d-4998-b22b-87b2aed6ba8f\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ng5vb" Oct 30 18:45:29.843301 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:29.843183 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1ef57ac-d33d-4998-b22b-87b2aed6ba8f-workload-certs\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ng5vb\" (UID: \"f1ef57ac-d33d-4998-b22b-87b2aed6ba8f\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ng5vb" Oct 30 18:45:29.846743 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:29.846720 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1ef57ac-d33d-4998-b22b-87b2aed6ba8f-istio-podinfo\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ng5vb\" (UID: \"f1ef57ac-d33d-4998-b22b-87b2aed6ba8f\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ng5vb" Oct 30 18:45:29.847580 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:29.847556 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1ef57ac-d33d-4998-b22b-87b2aed6ba8f-istio-envoy\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ng5vb\" (UID: \"f1ef57ac-d33d-4998-b22b-87b2aed6ba8f\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ng5vb" Oct 30 18:45:29.854550 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:29.854503 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1ef57ac-d33d-4998-b22b-87b2aed6ba8f-istio-token\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ng5vb\" (UID: \"f1ef57ac-d33d-4998-b22b-87b2aed6ba8f\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ng5vb" Oct 30 18:45:29.858164 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:29.854799 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb686\" (UniqueName: \"kubernetes.io/projected/f1ef57ac-d33d-4998-b22b-87b2aed6ba8f-kube-api-access-zb686\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ng5vb\" (UID: \"f1ef57ac-d33d-4998-b22b-87b2aed6ba8f\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ng5vb" Oct 30 18:45:29.996169 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:29.995364 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ng5vb" Oct 30 18:45:30.048574 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:30.048536 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:30.280170 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:30.280050 6960 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-m5wdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Oct 30 18:45:30.280170 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:30.280105 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-m5wdn" podUID="6178626a-4928-40ee-b26a-362f39ee1965" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Oct 30 18:45:30.330289 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:30.312999 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:30.441033 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:30.440897 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ng5vb"] Oct 30 18:45:30.442200 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:45:30.442137 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1ef57ac_d33d_4998_b22b_87b2aed6ba8f.slice/crio-a5920a594f4c8ede1e71734b6b45c73d366e23ef439f7e1a27398913f26a147c WatchSource:0}: Error finding container a5920a594f4c8ede1e71734b6b45c73d366e23ef439f7e1a27398913f26a147c: Status 404 returned error can't find the container with id a5920a594f4c8ede1e71734b6b45c73d366e23ef439f7e1a27398913f26a147c Oct 30 18:45:30.445417 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:30.445389 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:45:30.445482 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:30.445456 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:45:30.445517 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:30.445487 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:45:31.104246 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:31.104207 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ng5vb" event={"ID":"f1ef57ac-d33d-4998-b22b-87b2aed6ba8f","Type":"ContainerStarted","Data":"d9cf2cfc5db588847d9733b447ef0f784016c43123d8b0908aaf6c5d668993dd"} Oct 30 18:45:31.104406 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:31.104263 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ng5vb" event={"ID":"f1ef57ac-d33d-4998-b22b-87b2aed6ba8f","Type":"ContainerStarted","Data":"a5920a594f4c8ede1e71734b6b45c73d366e23ef439f7e1a27398913f26a147c"} Oct 30 18:45:31.283683 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:31.283650 6960 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-m5wdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Oct 30 18:45:31.284268 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:31.284236 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-m5wdn" podUID="6178626a-4928-40ee-b26a-362f39ee1965" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Oct 30 18:45:31.437308 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:31.437231 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:31.998206 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:31.997914 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ng5vb" Oct 30 18:45:31.998206 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:31.998041 6960 patch_prober.go:29] interesting pod/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ng5vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Oct 30 18:45:31.998206 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:31.998089 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ng5vb" podUID="f1ef57ac-d33d-4998-b22b-87b2aed6ba8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Oct 30 18:45:32.287230 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:32.286558 6960 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-m5wdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Oct 30 18:45:32.287230 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:32.286628 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-m5wdn" podUID="6178626a-4928-40ee-b26a-362f39ee1965" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Oct 30 18:45:32.771170 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:32.764236 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:33.000171 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:32.997606 6960 patch_prober.go:29] interesting pod/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ng5vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Oct 30 18:45:33.000171 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:32.997689 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ng5vb" podUID="f1ef57ac-d33d-4998-b22b-87b2aed6ba8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Oct 30 18:45:33.281168 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:33.280560 6960 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-m5wdn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Oct 30 18:45:33.281168 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:33.280620 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-m5wdn" podUID="6178626a-4928-40ee-b26a-362f39ee1965" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Oct 30 18:45:33.968735 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:33.968679 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:33.996829 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:33.996758 6960 patch_prober.go:29] interesting pod/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ng5vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Oct 30 18:45:33.997186 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:33.996840 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ng5vb" podUID="f1ef57ac-d33d-4998-b22b-87b2aed6ba8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Oct 30 18:45:34.577288 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:34.577254 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-m5wdn" Oct 30 18:45:34.578808 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:34.578784 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-m5wdn" Oct 30 18:45:34.586506 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:34.586481 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-m5wdn" Oct 30 18:45:34.609353 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:34.609304 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ng5vb" podStartSLOduration=5.609291033 podStartE2EDuration="5.609291033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:45:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:31.128068604 +0000 UTC m=+698.657099739" watchObservedRunningTime="2026-10-30 18:45:34.609291033 +0000 UTC m=+702.138322170" Oct 30 18:45:34.996057 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:34.995965 6960 patch_prober.go:29] interesting pod/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ng5vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Oct 30 18:45:34.996057 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:34.996021 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ng5vb" podUID="f1ef57ac-d33d-4998-b22b-87b2aed6ba8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Oct 30 18:45:35.106237 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:35.106205 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:35.998912 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:35.998872 6960 patch_prober.go:29] interesting pod/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ng5vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Oct 30 18:45:35.999486 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:35.998930 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ng5vb" podUID="f1ef57ac-d33d-4998-b22b-87b2aed6ba8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Oct 30 18:45:36.450544 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:36.450514 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:36.996702 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:36.996666 6960 patch_prober.go:29] interesting pod/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ng5vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Oct 30 18:45:36.996997 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:36.996968 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ng5vb" podUID="f1ef57ac-d33d-4998-b22b-87b2aed6ba8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Oct 30 18:45:37.688176 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:37.688123 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:38.595979 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:38.595930 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ng5vb" Oct 30 18:45:38.596474 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:38.596450 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ng5vb" Oct 30 18:45:38.597435 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:38.597415 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ng5vb" Oct 30 18:45:38.606490 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:38.606466 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:39.608602 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:39.608560 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:40.489618 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:40.489575 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:41.412562 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:41.412520 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:42.210483 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:42.210442 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:43.107388 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:43.107352 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:43.919185 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:43.919137 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:44.897881 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:44.897830 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:45.857312 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:45.857278 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:46.797387 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:46.797349 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:47.351960 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:47.351920 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-n9mh9"] Oct 30 18:45:47.357934 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:47.357910 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-n9mh9" Oct 30 18:45:47.375993 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:47.375969 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-84c32a-openshift-default-dockercfg-k6dxc\"" Oct 30 18:45:47.379095 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:47.379066 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-n9mh9"] Oct 30 18:45:47.441168 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:47.441115 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hnlz\" (UniqueName: \"kubernetes.io/projected/6eaab09d-e908-4663-9793-30cff7591dfb-kube-api-access-4hnlz\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-n9mh9\" (UID: \"6eaab09d-e908-4663-9793-30cff7591dfb\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-n9mh9" Oct 30 18:45:47.441320 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:47.441248 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6eaab09d-e908-4663-9793-30cff7591dfb-istiod-ca-cert\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-n9mh9\" (UID: \"6eaab09d-e908-4663-9793-30cff7591dfb\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-n9mh9" Oct 30 18:45:47.441320 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:47.441305 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eaab09d-e908-4663-9793-30cff7591dfb-credential-socket\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-n9mh9\" (UID: \"6eaab09d-e908-4663-9793-30cff7591dfb\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-n9mh9" Oct 30 18:45:47.441394 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:47.441343 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6eaab09d-e908-4663-9793-30cff7591dfb-istio-podinfo\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-n9mh9\" (UID: \"6eaab09d-e908-4663-9793-30cff7591dfb\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-n9mh9" Oct 30 18:45:47.441394 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:47.441367 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6eaab09d-e908-4663-9793-30cff7591dfb-istio-data\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-n9mh9\" (UID: \"6eaab09d-e908-4663-9793-30cff7591dfb\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-n9mh9" Oct 30 18:45:47.441459 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:47.441421 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6eaab09d-e908-4663-9793-30cff7591dfb-istio-envoy\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-n9mh9\" (UID: \"6eaab09d-e908-4663-9793-30cff7591dfb\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-n9mh9" Oct 30 18:45:47.441459 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:47.441444 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6eaab09d-e908-4663-9793-30cff7591dfb-istio-token\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-n9mh9\" (UID: \"6eaab09d-e908-4663-9793-30cff7591dfb\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-n9mh9" Oct 30 18:45:47.441525 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:47.441465 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6eaab09d-e908-4663-9793-30cff7591dfb-workload-certs\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-n9mh9\" (UID: \"6eaab09d-e908-4663-9793-30cff7591dfb\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-n9mh9" Oct 30 18:45:47.441600 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:47.441589 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eaab09d-e908-4663-9793-30cff7591dfb-workload-socket\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-n9mh9\" (UID: \"6eaab09d-e908-4663-9793-30cff7591dfb\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-n9mh9" Oct 30 18:45:47.542550 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:47.542516 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eaab09d-e908-4663-9793-30cff7591dfb-credential-socket\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-n9mh9\" (UID: \"6eaab09d-e908-4663-9793-30cff7591dfb\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-n9mh9" Oct 30 18:45:47.542726 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:47.542562 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6eaab09d-e908-4663-9793-30cff7591dfb-istio-podinfo\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-n9mh9\" (UID: \"6eaab09d-e908-4663-9793-30cff7591dfb\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-n9mh9" Oct 30 18:45:47.542726 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:47.542594 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6eaab09d-e908-4663-9793-30cff7591dfb-istio-data\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-n9mh9\" (UID: \"6eaab09d-e908-4663-9793-30cff7591dfb\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-n9mh9" Oct 30 18:45:47.542726 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:47.542619 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6eaab09d-e908-4663-9793-30cff7591dfb-istio-envoy\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-n9mh9\" (UID: \"6eaab09d-e908-4663-9793-30cff7591dfb\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-n9mh9" Oct 30 18:45:47.542726 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:47.542647 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6eaab09d-e908-4663-9793-30cff7591dfb-istio-token\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-n9mh9\" (UID: \"6eaab09d-e908-4663-9793-30cff7591dfb\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-n9mh9" Oct 30 18:45:47.542726 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:47.542677 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6eaab09d-e908-4663-9793-30cff7591dfb-workload-certs\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-n9mh9\" (UID: \"6eaab09d-e908-4663-9793-30cff7591dfb\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-n9mh9" Oct 30 18:45:47.543012 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:47.542797 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eaab09d-e908-4663-9793-30cff7591dfb-workload-socket\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-n9mh9\" (UID: \"6eaab09d-e908-4663-9793-30cff7591dfb\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-n9mh9" Oct 30 18:45:47.543012 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:47.542840 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hnlz\" (UniqueName: \"kubernetes.io/projected/6eaab09d-e908-4663-9793-30cff7591dfb-kube-api-access-4hnlz\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-n9mh9\" (UID: \"6eaab09d-e908-4663-9793-30cff7591dfb\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-n9mh9" Oct 30 18:45:47.543012 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:47.542905 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6eaab09d-e908-4663-9793-30cff7591dfb-istiod-ca-cert\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-n9mh9\" (UID: \"6eaab09d-e908-4663-9793-30cff7591dfb\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-n9mh9" Oct 30 18:45:47.543187 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:47.543002 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eaab09d-e908-4663-9793-30cff7591dfb-credential-socket\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-n9mh9\" (UID: \"6eaab09d-e908-4663-9793-30cff7591dfb\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-n9mh9" Oct 30 18:45:47.543187 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:47.543042 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6eaab09d-e908-4663-9793-30cff7591dfb-istio-data\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-n9mh9\" (UID: \"6eaab09d-e908-4663-9793-30cff7591dfb\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-n9mh9" Oct 30 18:45:47.543310 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:47.543261 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eaab09d-e908-4663-9793-30cff7591dfb-workload-socket\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-n9mh9\" (UID: \"6eaab09d-e908-4663-9793-30cff7591dfb\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-n9mh9" Oct 30 18:45:47.543382 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:47.543360 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6eaab09d-e908-4663-9793-30cff7591dfb-workload-certs\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-n9mh9\" (UID: \"6eaab09d-e908-4663-9793-30cff7591dfb\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-n9mh9" Oct 30 18:45:47.543616 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:47.543594 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6eaab09d-e908-4663-9793-30cff7591dfb-istiod-ca-cert\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-n9mh9\" (UID: \"6eaab09d-e908-4663-9793-30cff7591dfb\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-n9mh9" Oct 30 18:45:47.545050 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:47.545021 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6eaab09d-e908-4663-9793-30cff7591dfb-istio-envoy\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-n9mh9\" (UID: \"6eaab09d-e908-4663-9793-30cff7591dfb\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-n9mh9" Oct 30 18:45:47.545557 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:47.545536 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6eaab09d-e908-4663-9793-30cff7591dfb-istio-podinfo\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-n9mh9\" (UID: \"6eaab09d-e908-4663-9793-30cff7591dfb\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-n9mh9" Oct 30 18:45:47.562432 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:47.562397 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6eaab09d-e908-4663-9793-30cff7591dfb-istio-token\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-n9mh9\" (UID: \"6eaab09d-e908-4663-9793-30cff7591dfb\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-n9mh9" Oct 30 18:45:47.566473 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:47.566452 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hnlz\" (UniqueName: \"kubernetes.io/projected/6eaab09d-e908-4663-9793-30cff7591dfb-kube-api-access-4hnlz\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-n9mh9\" (UID: \"6eaab09d-e908-4663-9793-30cff7591dfb\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-n9mh9" Oct 30 18:45:47.671321 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:47.671235 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-n9mh9" Oct 30 18:45:47.927582 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:47.927504 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:47.950809 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:47.950790 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-n9mh9"] Oct 30 18:45:47.952553 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:45:47.952522 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6eaab09d_e908_4663_9793_30cff7591dfb.slice/crio-0344cdebe400b85c8489135c1339a7b8c80246d30b4943d3fb2f92ee365a34f7 WatchSource:0}: Error finding container 0344cdebe400b85c8489135c1339a7b8c80246d30b4943d3fb2f92ee365a34f7: Status 404 returned error can't find the container with id 0344cdebe400b85c8489135c1339a7b8c80246d30b4943d3fb2f92ee365a34f7 Oct 30 18:45:47.955066 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:47.955037 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:45:47.955203 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:47.955098 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:45:47.955203 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:47.955124 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:45:48.212396 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:48.212311 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-n9mh9" event={"ID":"6eaab09d-e908-4663-9793-30cff7591dfb","Type":"ContainerStarted","Data":"efb91e4d840cdee5f09dd66ad3266a4675fbf390be43ebefd7e21c85c9fca6bd"} Oct 30 18:45:48.212396 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:48.212359 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-n9mh9" event={"ID":"6eaab09d-e908-4663-9793-30cff7591dfb","Type":"ContainerStarted","Data":"0344cdebe400b85c8489135c1339a7b8c80246d30b4943d3fb2f92ee365a34f7"} Oct 30 18:45:48.238074 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:48.238006 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-n9mh9" podStartSLOduration=1.2379697649999999 podStartE2EDuration="1.237969765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:45:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:48.237448434 +0000 UTC m=+715.766479575" watchObservedRunningTime="2026-10-30 18:45:48.237969765 +0000 UTC m=+715.767000900" Oct 30 18:45:48.672032 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:48.672001 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-n9mh9" Oct 30 18:45:48.896279 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:48.896226 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:49.169736 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:49.169664 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:49.287857 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:49.287823 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:49.647008 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:49.646976 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:49.677204 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:49.676631 6960 patch_prober.go:29] interesting pod/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-n9mh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 18:45:49.677204 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:49.676700 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-n9mh9" podUID="6eaab09d-e908-4663-9793-30cff7591dfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 18:45:49.841636 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:49.841601 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:50.242739 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:50.242707 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:50.425359 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:50.425323 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:50.627303 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:50.627267 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:50.679283 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:50.679238 6960 patch_prober.go:29] interesting pod/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-n9mh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 18:45:50.679447 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:50.679298 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-n9mh9" podUID="6eaab09d-e908-4663-9793-30cff7591dfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 18:45:51.188552 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:51.188511 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:51.383307 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:51.383217 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:51.613859 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:51.613824 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dslrt"] Oct 30 18:45:51.617791 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:51.617775 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dslrt" Oct 30 18:45:51.667835 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:51.665979 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dslrt"] Oct 30 18:45:51.696570 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:51.696200 6960 patch_prober.go:29] interesting pod/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-n9mh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 18:45:51.696570 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:51.696242 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-n9mh9" podUID="6eaab09d-e908-4663-9793-30cff7591dfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 18:45:51.719457 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:51.719422 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xz5st\" (UniqueName: \"kubernetes.io/projected/93b3f3a7-cd85-4996-88fe-7762155b3241-kube-api-access-xz5st\") pod \"maas-default-gateway-openshift-default-855f9574cf-dslrt\" (UID: \"93b3f3a7-cd85-4996-88fe-7762155b3241\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dslrt" Oct 30 18:45:51.719627 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:51.719482 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93b3f3a7-cd85-4996-88fe-7762155b3241-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-dslrt\" (UID: \"93b3f3a7-cd85-4996-88fe-7762155b3241\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dslrt" Oct 30 18:45:51.719627 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:51.719578 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93b3f3a7-cd85-4996-88fe-7762155b3241-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-dslrt\" (UID: \"93b3f3a7-cd85-4996-88fe-7762155b3241\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dslrt" Oct 30 18:45:51.719748 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:51.719657 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93b3f3a7-cd85-4996-88fe-7762155b3241-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-dslrt\" (UID: \"93b3f3a7-cd85-4996-88fe-7762155b3241\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dslrt" Oct 30 18:45:51.719748 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:51.719703 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93b3f3a7-cd85-4996-88fe-7762155b3241-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-dslrt\" (UID: \"93b3f3a7-cd85-4996-88fe-7762155b3241\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dslrt" Oct 30 18:45:51.719748 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:51.719731 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93b3f3a7-cd85-4996-88fe-7762155b3241-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-dslrt\" (UID: \"93b3f3a7-cd85-4996-88fe-7762155b3241\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dslrt" Oct 30 18:45:51.719902 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:51.719765 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93b3f3a7-cd85-4996-88fe-7762155b3241-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-dslrt\" (UID: \"93b3f3a7-cd85-4996-88fe-7762155b3241\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dslrt" Oct 30 18:45:51.719902 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:51.719825 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93b3f3a7-cd85-4996-88fe-7762155b3241-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-dslrt\" (UID: \"93b3f3a7-cd85-4996-88fe-7762155b3241\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dslrt" Oct 30 18:45:51.719902 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:51.719875 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93b3f3a7-cd85-4996-88fe-7762155b3241-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-dslrt\" (UID: \"93b3f3a7-cd85-4996-88fe-7762155b3241\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dslrt" Oct 30 18:45:51.821418 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:51.821388 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xz5st\" (UniqueName: \"kubernetes.io/projected/93b3f3a7-cd85-4996-88fe-7762155b3241-kube-api-access-xz5st\") pod \"maas-default-gateway-openshift-default-855f9574cf-dslrt\" (UID: \"93b3f3a7-cd85-4996-88fe-7762155b3241\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dslrt" Oct 30 18:45:51.821418 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:51.821421 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93b3f3a7-cd85-4996-88fe-7762155b3241-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-dslrt\" (UID: \"93b3f3a7-cd85-4996-88fe-7762155b3241\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dslrt" Oct 30 18:45:51.821642 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:51.821450 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93b3f3a7-cd85-4996-88fe-7762155b3241-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-dslrt\" (UID: \"93b3f3a7-cd85-4996-88fe-7762155b3241\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dslrt" Oct 30 18:45:51.821642 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:51.821497 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93b3f3a7-cd85-4996-88fe-7762155b3241-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-dslrt\" (UID: \"93b3f3a7-cd85-4996-88fe-7762155b3241\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dslrt" Oct 30 18:45:51.821642 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:51.821520 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93b3f3a7-cd85-4996-88fe-7762155b3241-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-dslrt\" (UID: \"93b3f3a7-cd85-4996-88fe-7762155b3241\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dslrt" Oct 30 18:45:51.821642 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:51.821536 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93b3f3a7-cd85-4996-88fe-7762155b3241-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-dslrt\" (UID: \"93b3f3a7-cd85-4996-88fe-7762155b3241\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dslrt" Oct 30 18:45:51.821642 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:51.821558 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93b3f3a7-cd85-4996-88fe-7762155b3241-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-dslrt\" (UID: \"93b3f3a7-cd85-4996-88fe-7762155b3241\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dslrt" Oct 30 18:45:51.821642 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:51.821595 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93b3f3a7-cd85-4996-88fe-7762155b3241-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-dslrt\" (UID: \"93b3f3a7-cd85-4996-88fe-7762155b3241\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dslrt" Oct 30 18:45:51.821642 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:51.821613 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93b3f3a7-cd85-4996-88fe-7762155b3241-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-dslrt\" (UID: \"93b3f3a7-cd85-4996-88fe-7762155b3241\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dslrt" Oct 30 18:45:51.821968 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:51.821941 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93b3f3a7-cd85-4996-88fe-7762155b3241-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-dslrt\" (UID: \"93b3f3a7-cd85-4996-88fe-7762155b3241\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dslrt" Oct 30 18:45:51.822303 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:51.822234 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93b3f3a7-cd85-4996-88fe-7762155b3241-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-dslrt\" (UID: \"93b3f3a7-cd85-4996-88fe-7762155b3241\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dslrt" Oct 30 18:45:51.822466 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:51.822444 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93b3f3a7-cd85-4996-88fe-7762155b3241-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-dslrt\" (UID: \"93b3f3a7-cd85-4996-88fe-7762155b3241\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dslrt" Oct 30 18:45:51.822532 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:51.822512 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93b3f3a7-cd85-4996-88fe-7762155b3241-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-dslrt\" (UID: \"93b3f3a7-cd85-4996-88fe-7762155b3241\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dslrt" Oct 30 18:45:51.822635 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:51.822617 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93b3f3a7-cd85-4996-88fe-7762155b3241-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-dslrt\" (UID: \"93b3f3a7-cd85-4996-88fe-7762155b3241\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dslrt" Oct 30 18:45:51.824319 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:51.824299 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93b3f3a7-cd85-4996-88fe-7762155b3241-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-dslrt\" (UID: \"93b3f3a7-cd85-4996-88fe-7762155b3241\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dslrt" Oct 30 18:45:51.824684 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:51.824665 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93b3f3a7-cd85-4996-88fe-7762155b3241-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-dslrt\" (UID: \"93b3f3a7-cd85-4996-88fe-7762155b3241\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dslrt" Oct 30 18:45:51.881125 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:51.881043 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:51.882584 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:51.882448 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xz5st\" (UniqueName: \"kubernetes.io/projected/93b3f3a7-cd85-4996-88fe-7762155b3241-kube-api-access-xz5st\") pod \"maas-default-gateway-openshift-default-855f9574cf-dslrt\" (UID: \"93b3f3a7-cd85-4996-88fe-7762155b3241\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dslrt" Oct 30 18:45:51.882584 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:51.882528 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93b3f3a7-cd85-4996-88fe-7762155b3241-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-dslrt\" (UID: \"93b3f3a7-cd85-4996-88fe-7762155b3241\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dslrt" Oct 30 18:45:51.932789 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:51.932765 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dslrt" Oct 30 18:45:52.284622 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:52.283569 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dslrt"] Oct 30 18:45:52.294355 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:52.293019 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:45:52.294355 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:52.293107 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:45:52.294355 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:52.293184 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:45:52.685168 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:52.684477 6960 patch_prober.go:29] interesting pod/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-n9mh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 18:45:52.685168 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:52.684536 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-n9mh9" podUID="6eaab09d-e908-4663-9793-30cff7591dfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 18:45:52.978218 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:52.974810 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:53.244027 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:53.243949 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dslrt" event={"ID":"93b3f3a7-cd85-4996-88fe-7762155b3241","Type":"ContainerStarted","Data":"77b72cadc1cfa7421c7124a1217170a2273733d337fa610d5e345d7aa1646de6"} Oct 30 18:45:53.244027 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:53.243998 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dslrt" event={"ID":"93b3f3a7-cd85-4996-88fe-7762155b3241","Type":"ContainerStarted","Data":"961cd7a2ec490855515af8ebbca45836e4bfff084770b8aaa185450d0375f12b"} Oct 30 18:45:53.275171 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:53.274532 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dslrt" podStartSLOduration=2.274512895 podStartE2EDuration="2.274512895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:45:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:53.274436082 +0000 UTC m=+720.803467221" watchObservedRunningTime="2026-10-30 18:45:53.274512895 +0000 UTC m=+720.803544036" Oct 30 18:45:53.673183 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:53.672566 6960 patch_prober.go:29] interesting pod/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-n9mh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 18:45:53.673183 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:53.672630 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-n9mh9" podUID="6eaab09d-e908-4663-9793-30cff7591dfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 18:45:53.933390 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:53.933311 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dslrt" Oct 30 18:45:53.934915 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:53.934889 6960 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-dslrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 30 18:45:53.935058 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:53.934938 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dslrt" podUID="93b3f3a7-cd85-4996-88fe-7762155b3241" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 30 18:45:53.937361 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:45:53.937337 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 18:45:53.973168 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:53.972348 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:54.845854 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:54.845820 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-n9mh9" Oct 30 18:45:54.846480 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:54.846400 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-n9mh9" Oct 30 18:45:54.847254 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:54.847234 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-n9mh9" Oct 30 18:45:54.882661 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:54.882621 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:54.938703 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:54.933309 6960 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-dslrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 30 18:45:54.938703 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:54.933380 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dslrt" podUID="93b3f3a7-cd85-4996-88fe-7762155b3241" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 30 18:45:55.535223 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:55.535188 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:55.933937 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:55.933904 6960 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-dslrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 30 18:45:55.934103 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:55.933956 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dslrt" podUID="93b3f3a7-cd85-4996-88fe-7762155b3241" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 30 18:45:56.024450 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:56.024414 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:56.150607 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:56.150573 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:56.933651 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:56.933619 6960 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-dslrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 30 18:45:56.933833 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:56.933672 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dslrt" podUID="93b3f3a7-cd85-4996-88fe-7762155b3241" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 30 18:45:57.033029 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:57.032995 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:57.933381 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:57.933354 6960 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-dslrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 30 18:45:57.933547 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:57.933412 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dslrt" podUID="93b3f3a7-cd85-4996-88fe-7762155b3241" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 30 18:45:58.048745 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:58.048702 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:58.415849 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:58.415816 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:58.596095 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:58.596056 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:58.933509 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:58.933460 6960 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-dslrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 30 18:45:58.933693 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:58.933533 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dslrt" podUID="93b3f3a7-cd85-4996-88fe-7762155b3241" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 30 18:45:59.009733 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:59.009702 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:59.933070 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:59.933033 6960 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-dslrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 30 18:45:59.933486 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:45:59.933107 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dslrt" podUID="93b3f3a7-cd85-4996-88fe-7762155b3241" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 30 18:46:00.007031 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:00.006989 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:00.939393 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:00.939351 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:01.360901 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:01.360857 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dslrt" Oct 30 18:46:01.362350 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:01.362323 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dslrt" Oct 30 18:46:01.363026 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:01.363004 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dslrt" Oct 30 18:46:01.816561 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:01.816529 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:02.964834 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:02.964802 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:04.200227 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:04.199120 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:05.499169 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:05.496111 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:06.694940 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:06.694905 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:07.491108 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:07.491074 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:08.298452 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:08.298405 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:09.218174 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:09.218112 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:10.107397 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:10.107358 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:10.865549 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:10.865517 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:11.658970 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:11.658938 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:11.769231 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:11.769198 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:12.652812 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:12.652772 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:13.560799 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:13.560768 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:14.459650 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:14.459613 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:14.568999 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:14.568940 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:14.714452 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:14.714375 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-m4drm"] Oct 30 18:46:14.718424 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:14.718404 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-m4drm" Oct 30 18:46:14.772084 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:14.772042 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-m4drm"] Oct 30 18:46:14.812302 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:14.812266 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684-credential-socket\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-m4drm\" (UID: \"e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-m4drm" Oct 30 18:46:14.812457 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:14.812325 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684-istio-token\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-m4drm\" (UID: \"e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-m4drm" Oct 30 18:46:14.812457 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:14.812437 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684-istio-data\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-m4drm\" (UID: \"e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-m4drm" Oct 30 18:46:14.812530 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:14.812511 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684-workload-certs\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-m4drm\" (UID: \"e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-m4drm" Oct 30 18:46:14.812565 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:14.812546 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684-istio-envoy\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-m4drm\" (UID: \"e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-m4drm" Oct 30 18:46:14.812603 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:14.812592 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684-istiod-ca-cert\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-m4drm\" (UID: \"e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-m4drm" Oct 30 18:46:14.812639 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:14.812611 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684-workload-socket\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-m4drm\" (UID: \"e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-m4drm" Oct 30 18:46:14.812639 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:14.812631 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684-istio-podinfo\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-m4drm\" (UID: \"e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-m4drm" Oct 30 18:46:14.812701 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:14.812689 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47vjr\" (UniqueName: \"kubernetes.io/projected/e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684-kube-api-access-47vjr\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-m4drm\" (UID: \"e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-m4drm" Oct 30 18:46:14.913823 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:14.913784 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684-istio-data\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-m4drm\" (UID: \"e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-m4drm" Oct 30 18:46:14.914011 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:14.913846 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684-workload-certs\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-m4drm\" (UID: \"e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-m4drm" Oct 30 18:46:14.914011 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:14.913888 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684-istio-envoy\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-m4drm\" (UID: \"e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-m4drm" Oct 30 18:46:14.914011 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:14.913919 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684-istiod-ca-cert\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-m4drm\" (UID: \"e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-m4drm" Oct 30 18:46:14.914011 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:14.913945 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684-workload-socket\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-m4drm\" (UID: \"e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-m4drm" Oct 30 18:46:14.914011 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:14.913969 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684-istio-podinfo\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-m4drm\" (UID: \"e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-m4drm" Oct 30 18:46:14.914011 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:14.914000 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47vjr\" (UniqueName: \"kubernetes.io/projected/e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684-kube-api-access-47vjr\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-m4drm\" (UID: \"e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-m4drm" Oct 30 18:46:14.914343 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:14.914089 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684-credential-socket\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-m4drm\" (UID: \"e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-m4drm" Oct 30 18:46:14.914343 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:14.914184 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684-istio-token\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-m4drm\" (UID: \"e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-m4drm" Oct 30 18:46:14.914449 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:14.914349 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684-istio-data\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-m4drm\" (UID: \"e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-m4drm" Oct 30 18:46:14.914449 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:14.914358 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684-workload-certs\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-m4drm\" (UID: \"e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-m4drm" Oct 30 18:46:14.914632 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:14.914604 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684-credential-socket\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-m4drm\" (UID: \"e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-m4drm" Oct 30 18:46:14.914761 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:14.914704 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684-workload-socket\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-m4drm\" (UID: \"e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-m4drm" Oct 30 18:46:14.914820 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:14.914760 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684-istiod-ca-cert\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-m4drm\" (UID: \"e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-m4drm" Oct 30 18:46:14.916731 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:14.916700 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684-istio-podinfo\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-m4drm\" (UID: \"e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-m4drm" Oct 30 18:46:14.916731 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:14.916703 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684-istio-envoy\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-m4drm\" (UID: \"e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-m4drm" Oct 30 18:46:14.926974 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:14.926946 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47vjr\" (UniqueName: \"kubernetes.io/projected/e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684-kube-api-access-47vjr\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-m4drm\" (UID: \"e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-m4drm" Oct 30 18:46:14.929988 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:14.929969 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684-istio-token\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-m4drm\" (UID: \"e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-m4drm" Oct 30 18:46:15.033094 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:15.033007 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-m4drm" Oct 30 18:46:15.314803 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:15.314775 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-m4drm"] Oct 30 18:46:15.322159 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:15.321817 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:46:15.322159 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:15.321908 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:46:15.322159 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:15.321950 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:46:15.415625 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:15.415583 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-m4drm" event={"ID":"e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684","Type":"ContainerStarted","Data":"faca51b8559e821ce29a2bd6f20555950e5384cd03d6e13aef06c772e400f514"} Oct 30 18:46:15.415774 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:15.415655 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-m4drm" event={"ID":"e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684","Type":"ContainerStarted","Data":"7cfba0a528c82321e6c1e149af0a8c80ff434aff0d0defbea042101638a4f0b6"} Oct 30 18:46:15.599629 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:15.599541 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:16.033282 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:16.033197 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-m4drm" Oct 30 18:46:16.034777 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:16.034752 6960 patch_prober.go:29] interesting pod/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-m4drm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 18:46:16.034924 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:16.034798 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-m4drm" podUID="e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 18:46:16.395330 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:16.395295 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:17.033410 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:17.033378 6960 patch_prober.go:29] interesting pod/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-m4drm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 18:46:17.033793 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:17.033433 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-m4drm" podUID="e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 18:46:17.275107 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:17.274596 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:18.033900 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:18.033871 6960 patch_prober.go:29] interesting pod/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-m4drm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 18:46:18.034320 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:18.033948 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-m4drm" podUID="e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 18:46:18.380360 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:18.380311 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:19.033540 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:19.033504 6960 patch_prober.go:29] interesting pod/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-m4drm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 18:46:19.033735 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:19.033577 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-m4drm" podUID="e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 18:46:19.241469 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:19.241431 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:20.034418 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:20.034383 6960 patch_prober.go:29] interesting pod/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-m4drm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 18:46:20.034588 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:20.034442 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-m4drm" podUID="e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 18:46:20.216072 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:20.216037 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:21.034021 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:21.033987 6960 patch_prober.go:29] interesting pod/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-m4drm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 18:46:21.034418 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:21.034043 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-m4drm" podUID="e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 18:46:21.135103 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:21.135067 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:22.226649 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:22.226612 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:22.369262 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:22.369238 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-m4drm" Oct 30 18:46:22.369664 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:22.369641 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-m4drm" Oct 30 18:46:22.370464 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:22.370444 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-m4drm" Oct 30 18:46:22.395503 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:22.395446 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-m4drm" podStartSLOduration=8.395430963 podStartE2EDuration="8.395430963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:46:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:46:15.453470815 +0000 UTC m=+742.982501954" watchObservedRunningTime="2026-10-30 18:46:22.395430963 +0000 UTC m=+749.924462104" Oct 30 18:46:23.293223 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:23.293190 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:23.933505 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:46:23.933462 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 18:46:24.034161 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:24.034100 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:24.171053 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:24.171013 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:25.107084 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:25.107052 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:25.862177 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:25.862113 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:26.505762 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:26.505730 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:26.782103 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:26.782011 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:27.073746 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:27.073715 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-phklq"] Oct 30 18:46:27.079078 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:27.079052 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-phklq" Oct 30 18:46:27.096461 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:27.096431 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-phklq"] Oct 30 18:46:27.174250 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:27.174222 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67bc209e-b148-4464-912e-f1ba5f372c50-credential-socket\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-phklq\" (UID: \"67bc209e-b148-4464-912e-f1ba5f372c50\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-phklq" Oct 30 18:46:27.174431 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:27.174261 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67bc209e-b148-4464-912e-f1ba5f372c50-istio-podinfo\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-phklq\" (UID: \"67bc209e-b148-4464-912e-f1ba5f372c50\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-phklq" Oct 30 18:46:27.174431 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:27.174337 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67bc209e-b148-4464-912e-f1ba5f372c50-istiod-ca-cert\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-phklq\" (UID: \"67bc209e-b148-4464-912e-f1ba5f372c50\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-phklq" Oct 30 18:46:27.174562 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:27.174434 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67bc209e-b148-4464-912e-f1ba5f372c50-istio-data\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-phklq\" (UID: \"67bc209e-b148-4464-912e-f1ba5f372c50\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-phklq" Oct 30 18:46:27.174562 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:27.174498 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67bc209e-b148-4464-912e-f1ba5f372c50-workload-certs\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-phklq\" (UID: \"67bc209e-b148-4464-912e-f1ba5f372c50\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-phklq" Oct 30 18:46:27.174645 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:27.174574 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pljbr\" (UniqueName: \"kubernetes.io/projected/67bc209e-b148-4464-912e-f1ba5f372c50-kube-api-access-pljbr\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-phklq\" (UID: \"67bc209e-b148-4464-912e-f1ba5f372c50\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-phklq" Oct 30 18:46:27.174645 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:27.174636 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67bc209e-b148-4464-912e-f1ba5f372c50-workload-socket\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-phklq\" (UID: \"67bc209e-b148-4464-912e-f1ba5f372c50\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-phklq" Oct 30 18:46:27.174727 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:27.174700 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67bc209e-b148-4464-912e-f1ba5f372c50-istio-envoy\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-phklq\" (UID: \"67bc209e-b148-4464-912e-f1ba5f372c50\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-phklq" Oct 30 18:46:27.174818 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:27.174800 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67bc209e-b148-4464-912e-f1ba5f372c50-istio-token\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-phklq\" (UID: \"67bc209e-b148-4464-912e-f1ba5f372c50\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-phklq" Oct 30 18:46:27.275734 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:27.275704 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67bc209e-b148-4464-912e-f1ba5f372c50-istio-envoy\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-phklq\" (UID: \"67bc209e-b148-4464-912e-f1ba5f372c50\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-phklq" Oct 30 18:46:27.275885 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:27.275768 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67bc209e-b148-4464-912e-f1ba5f372c50-istio-token\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-phklq\" (UID: \"67bc209e-b148-4464-912e-f1ba5f372c50\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-phklq" Oct 30 18:46:27.275885 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:27.275863 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67bc209e-b148-4464-912e-f1ba5f372c50-credential-socket\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-phklq\" (UID: \"67bc209e-b148-4464-912e-f1ba5f372c50\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-phklq" Oct 30 18:46:27.275986 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:27.275902 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67bc209e-b148-4464-912e-f1ba5f372c50-istio-podinfo\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-phklq\" (UID: \"67bc209e-b148-4464-912e-f1ba5f372c50\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-phklq" Oct 30 18:46:27.275986 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:27.275929 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67bc209e-b148-4464-912e-f1ba5f372c50-istiod-ca-cert\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-phklq\" (UID: \"67bc209e-b148-4464-912e-f1ba5f372c50\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-phklq" Oct 30 18:46:27.275986 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:27.275950 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67bc209e-b148-4464-912e-f1ba5f372c50-istio-data\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-phklq\" (UID: \"67bc209e-b148-4464-912e-f1ba5f372c50\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-phklq" Oct 30 18:46:27.276128 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:27.276007 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67bc209e-b148-4464-912e-f1ba5f372c50-workload-certs\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-phklq\" (UID: \"67bc209e-b148-4464-912e-f1ba5f372c50\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-phklq" Oct 30 18:46:27.276128 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:27.276058 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pljbr\" (UniqueName: \"kubernetes.io/projected/67bc209e-b148-4464-912e-f1ba5f372c50-kube-api-access-pljbr\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-phklq\" (UID: \"67bc209e-b148-4464-912e-f1ba5f372c50\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-phklq" Oct 30 18:46:27.276128 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:27.276114 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67bc209e-b148-4464-912e-f1ba5f372c50-workload-socket\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-phklq\" (UID: \"67bc209e-b148-4464-912e-f1ba5f372c50\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-phklq" Oct 30 18:46:27.276483 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:27.276462 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67bc209e-b148-4464-912e-f1ba5f372c50-workload-certs\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-phklq\" (UID: \"67bc209e-b148-4464-912e-f1ba5f372c50\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-phklq" Oct 30 18:46:27.276634 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:27.276607 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67bc209e-b148-4464-912e-f1ba5f372c50-istio-data\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-phklq\" (UID: \"67bc209e-b148-4464-912e-f1ba5f372c50\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-phklq" Oct 30 18:46:27.276765 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:27.276709 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67bc209e-b148-4464-912e-f1ba5f372c50-istiod-ca-cert\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-phklq\" (UID: \"67bc209e-b148-4464-912e-f1ba5f372c50\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-phklq" Oct 30 18:46:27.276825 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:27.276790 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67bc209e-b148-4464-912e-f1ba5f372c50-workload-socket\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-phklq\" (UID: \"67bc209e-b148-4464-912e-f1ba5f372c50\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-phklq" Oct 30 18:46:27.276876 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:27.276861 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67bc209e-b148-4464-912e-f1ba5f372c50-credential-socket\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-phklq\" (UID: \"67bc209e-b148-4464-912e-f1ba5f372c50\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-phklq" Oct 30 18:46:27.278172 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:27.278131 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67bc209e-b148-4464-912e-f1ba5f372c50-istio-envoy\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-phklq\" (UID: \"67bc209e-b148-4464-912e-f1ba5f372c50\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-phklq" Oct 30 18:46:27.278475 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:27.278459 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67bc209e-b148-4464-912e-f1ba5f372c50-istio-podinfo\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-phklq\" (UID: \"67bc209e-b148-4464-912e-f1ba5f372c50\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-phklq" Oct 30 18:46:27.288013 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:27.287992 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67bc209e-b148-4464-912e-f1ba5f372c50-istio-token\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-phklq\" (UID: \"67bc209e-b148-4464-912e-f1ba5f372c50\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-phklq" Oct 30 18:46:27.288189 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:27.288167 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pljbr\" (UniqueName: \"kubernetes.io/projected/67bc209e-b148-4464-912e-f1ba5f372c50-kube-api-access-pljbr\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-phklq\" (UID: \"67bc209e-b148-4464-912e-f1ba5f372c50\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-phklq" Oct 30 18:46:27.401097 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:27.401028 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-phklq" Oct 30 18:46:27.698417 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:27.698390 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:27.699625 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:27.699603 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-phklq"] Oct 30 18:46:27.701866 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:46:27.701842 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67bc209e_b148_4464_912e_f1ba5f372c50.slice/crio-a1c2286ea525632daa3dc36f44c93ac68cdcc5d0d73a13476797e8da7fbd13c3 WatchSource:0}: Error finding container a1c2286ea525632daa3dc36f44c93ac68cdcc5d0d73a13476797e8da7fbd13c3: Status 404 returned error can't find the container with id a1c2286ea525632daa3dc36f44c93ac68cdcc5d0d73a13476797e8da7fbd13c3 Oct 30 18:46:27.704178 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:27.704123 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:46:27.704271 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:27.704208 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:46:27.704271 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:27.704238 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:46:28.479134 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:28.479093 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:28.502384 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:28.499238 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-phklq" event={"ID":"67bc209e-b148-4464-912e-f1ba5f372c50","Type":"ContainerStarted","Data":"81fe2a9bc4b5feb43b828bfa531e8b17f3aa84c0e47f4e428856ea2a229749ec"} Oct 30 18:46:28.502384 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:28.499352 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-phklq" event={"ID":"67bc209e-b148-4464-912e-f1ba5f372c50","Type":"ContainerStarted","Data":"a1c2286ea525632daa3dc36f44c93ac68cdcc5d0d73a13476797e8da7fbd13c3"} Oct 30 18:46:28.526519 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:28.526465 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-phklq" podStartSLOduration=1.526450619 podStartE2EDuration="1.526450619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:46:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:46:28.524621354 +0000 UTC m=+756.053652496" watchObservedRunningTime="2026-10-30 18:46:28.526450619 +0000 UTC m=+756.055481756" Oct 30 18:46:29.285939 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:29.285842 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:29.401560 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:29.401522 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-phklq" Oct 30 18:46:29.403090 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:29.403062 6960 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-phklq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 18:46:29.403231 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:29.403119 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-phklq" podUID="67bc209e-b148-4464-912e-f1ba5f372c50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 18:46:29.690534 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:29.690502 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-aa6736/distinct-w1-aa6736-kserve-b86d7667b-smcbz"] Oct 30 18:46:29.694636 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:29.694612 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-aa6736/distinct-w1-aa6736-kserve-b86d7667b-smcbz" Oct 30 18:46:29.700919 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:29.700898 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-aa6736\"/\"distinct-w1-aa6736-kserve-self-signed-certs\"" Oct 30 18:46:29.703001 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:29.702978 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-aa6736/distinct-w1-aa6736-kserve-b86d7667b-smcbz"] Oct 30 18:46:29.705566 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:29.705543 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58c23db7-11cb-4ce2-a67e-fe698e89c71d-tmp-dir\") pod \"distinct-w1-aa6736-kserve-b86d7667b-smcbz\" (UID: \"58c23db7-11cb-4ce2-a67e-fe698e89c71d\") " pod="e2e-models-e2e-worker-w1-aa6736/distinct-w1-aa6736-kserve-b86d7667b-smcbz" Oct 30 18:46:29.706745 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:29.706722 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58c23db7-11cb-4ce2-a67e-fe698e89c71d-model-cache\") pod \"distinct-w1-aa6736-kserve-b86d7667b-smcbz\" (UID: \"58c23db7-11cb-4ce2-a67e-fe698e89c71d\") " pod="e2e-models-e2e-worker-w1-aa6736/distinct-w1-aa6736-kserve-b86d7667b-smcbz" Oct 30 18:46:29.706907 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:29.706886 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58c23db7-11cb-4ce2-a67e-fe698e89c71d-dshm\") pod \"distinct-w1-aa6736-kserve-b86d7667b-smcbz\" (UID: \"58c23db7-11cb-4ce2-a67e-fe698e89c71d\") " pod="e2e-models-e2e-worker-w1-aa6736/distinct-w1-aa6736-kserve-b86d7667b-smcbz" Oct 30 18:46:29.706994 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:29.706934 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58c23db7-11cb-4ce2-a67e-fe698e89c71d-tls-certs\") pod \"distinct-w1-aa6736-kserve-b86d7667b-smcbz\" (UID: \"58c23db7-11cb-4ce2-a67e-fe698e89c71d\") " pod="e2e-models-e2e-worker-w1-aa6736/distinct-w1-aa6736-kserve-b86d7667b-smcbz" Oct 30 18:46:29.706994 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:29.706971 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58c23db7-11cb-4ce2-a67e-fe698e89c71d-home\") pod \"distinct-w1-aa6736-kserve-b86d7667b-smcbz\" (UID: \"58c23db7-11cb-4ce2-a67e-fe698e89c71d\") " pod="e2e-models-e2e-worker-w1-aa6736/distinct-w1-aa6736-kserve-b86d7667b-smcbz" Oct 30 18:46:29.707105 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:29.707008 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hv4j5\" (UniqueName: \"kubernetes.io/projected/58c23db7-11cb-4ce2-a67e-fe698e89c71d-kube-api-access-hv4j5\") pod \"distinct-w1-aa6736-kserve-b86d7667b-smcbz\" (UID: \"58c23db7-11cb-4ce2-a67e-fe698e89c71d\") " pod="e2e-models-e2e-worker-w1-aa6736/distinct-w1-aa6736-kserve-b86d7667b-smcbz" Oct 30 18:46:29.738789 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:29.738753 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:29.810292 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:29.807866 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58c23db7-11cb-4ce2-a67e-fe698e89c71d-model-cache\") pod \"distinct-w1-aa6736-kserve-b86d7667b-smcbz\" (UID: \"58c23db7-11cb-4ce2-a67e-fe698e89c71d\") " pod="e2e-models-e2e-worker-w1-aa6736/distinct-w1-aa6736-kserve-b86d7667b-smcbz" Oct 30 18:46:29.810292 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:29.807914 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58c23db7-11cb-4ce2-a67e-fe698e89c71d-dshm\") pod \"distinct-w1-aa6736-kserve-b86d7667b-smcbz\" (UID: \"58c23db7-11cb-4ce2-a67e-fe698e89c71d\") " pod="e2e-models-e2e-worker-w1-aa6736/distinct-w1-aa6736-kserve-b86d7667b-smcbz" Oct 30 18:46:29.810292 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:29.807936 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58c23db7-11cb-4ce2-a67e-fe698e89c71d-tls-certs\") pod \"distinct-w1-aa6736-kserve-b86d7667b-smcbz\" (UID: \"58c23db7-11cb-4ce2-a67e-fe698e89c71d\") " pod="e2e-models-e2e-worker-w1-aa6736/distinct-w1-aa6736-kserve-b86d7667b-smcbz" Oct 30 18:46:29.810292 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:29.807959 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58c23db7-11cb-4ce2-a67e-fe698e89c71d-home\") pod \"distinct-w1-aa6736-kserve-b86d7667b-smcbz\" (UID: \"58c23db7-11cb-4ce2-a67e-fe698e89c71d\") " pod="e2e-models-e2e-worker-w1-aa6736/distinct-w1-aa6736-kserve-b86d7667b-smcbz" Oct 30 18:46:29.810292 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:29.807983 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hv4j5\" (UniqueName: \"kubernetes.io/projected/58c23db7-11cb-4ce2-a67e-fe698e89c71d-kube-api-access-hv4j5\") pod \"distinct-w1-aa6736-kserve-b86d7667b-smcbz\" (UID: \"58c23db7-11cb-4ce2-a67e-fe698e89c71d\") " pod="e2e-models-e2e-worker-w1-aa6736/distinct-w1-aa6736-kserve-b86d7667b-smcbz" Oct 30 18:46:29.810292 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:29.808024 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58c23db7-11cb-4ce2-a67e-fe698e89c71d-tmp-dir\") pod \"distinct-w1-aa6736-kserve-b86d7667b-smcbz\" (UID: \"58c23db7-11cb-4ce2-a67e-fe698e89c71d\") " pod="e2e-models-e2e-worker-w1-aa6736/distinct-w1-aa6736-kserve-b86d7667b-smcbz" Oct 30 18:46:29.810292 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:29.808640 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58c23db7-11cb-4ce2-a67e-fe698e89c71d-home\") pod \"distinct-w1-aa6736-kserve-b86d7667b-smcbz\" (UID: \"58c23db7-11cb-4ce2-a67e-fe698e89c71d\") " pod="e2e-models-e2e-worker-w1-aa6736/distinct-w1-aa6736-kserve-b86d7667b-smcbz" Oct 30 18:46:29.810292 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:29.808814 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58c23db7-11cb-4ce2-a67e-fe698e89c71d-model-cache\") pod \"distinct-w1-aa6736-kserve-b86d7667b-smcbz\" (UID: \"58c23db7-11cb-4ce2-a67e-fe698e89c71d\") " pod="e2e-models-e2e-worker-w1-aa6736/distinct-w1-aa6736-kserve-b86d7667b-smcbz" Oct 30 18:46:29.810292 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:29.808869 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58c23db7-11cb-4ce2-a67e-fe698e89c71d-tmp-dir\") pod \"distinct-w1-aa6736-kserve-b86d7667b-smcbz\" (UID: \"58c23db7-11cb-4ce2-a67e-fe698e89c71d\") " pod="e2e-models-e2e-worker-w1-aa6736/distinct-w1-aa6736-kserve-b86d7667b-smcbz" Oct 30 18:46:29.811897 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:29.811875 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58c23db7-11cb-4ce2-a67e-fe698e89c71d-tls-certs\") pod \"distinct-w1-aa6736-kserve-b86d7667b-smcbz\" (UID: \"58c23db7-11cb-4ce2-a67e-fe698e89c71d\") " pod="e2e-models-e2e-worker-w1-aa6736/distinct-w1-aa6736-kserve-b86d7667b-smcbz" Oct 30 18:46:29.816006 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:29.815976 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58c23db7-11cb-4ce2-a67e-fe698e89c71d-dshm\") pod \"distinct-w1-aa6736-kserve-b86d7667b-smcbz\" (UID: \"58c23db7-11cb-4ce2-a67e-fe698e89c71d\") " pod="e2e-models-e2e-worker-w1-aa6736/distinct-w1-aa6736-kserve-b86d7667b-smcbz" Oct 30 18:46:29.848977 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:29.848486 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hv4j5\" (UniqueName: \"kubernetes.io/projected/58c23db7-11cb-4ce2-a67e-fe698e89c71d-kube-api-access-hv4j5\") pod \"distinct-w1-aa6736-kserve-b86d7667b-smcbz\" (UID: \"58c23db7-11cb-4ce2-a67e-fe698e89c71d\") " pod="e2e-models-e2e-worker-w1-aa6736/distinct-w1-aa6736-kserve-b86d7667b-smcbz" Oct 30 18:46:30.030326 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:30.030250 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-aa6736/distinct-w1-aa6736-kserve-b86d7667b-smcbz" Oct 30 18:46:30.338306 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:30.338272 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:30.345972 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:46:30.345945 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod58c23db7_11cb_4ce2_a67e_fe698e89c71d.slice/crio-c677e600b5c27479c68933e741ce1c4f82a50125af51deb399fdb712049103cd WatchSource:0}: Error finding container c677e600b5c27479c68933e741ce1c4f82a50125af51deb399fdb712049103cd: Status 404 returned error can't find the container with id c677e600b5c27479c68933e741ce1c4f82a50125af51deb399fdb712049103cd Oct 30 18:46:30.346068 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:30.346025 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-aa6736/distinct-w1-aa6736-kserve-b86d7667b-smcbz"] Oct 30 18:46:30.401959 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:30.401912 6960 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-phklq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 18:46:30.402127 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:30.401981 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-phklq" podUID="67bc209e-b148-4464-912e-f1ba5f372c50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 18:46:30.511891 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:30.511858 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-aa6736/distinct-w1-aa6736-kserve-b86d7667b-smcbz" event={"ID":"58c23db7-11cb-4ce2-a67e-fe698e89c71d","Type":"ContainerStarted","Data":"c5915f27d8d9f6e341312f052cc2e232ecd2170eeb1f9fc05a374f08aa593681"} Oct 30 18:46:30.512095 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:30.512080 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-aa6736/distinct-w1-aa6736-kserve-b86d7667b-smcbz" event={"ID":"58c23db7-11cb-4ce2-a67e-fe698e89c71d","Type":"ContainerStarted","Data":"c677e600b5c27479c68933e741ce1c4f82a50125af51deb399fdb712049103cd"} Oct 30 18:46:30.532183 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:30.532020 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-aa6736/distinct-w1-aa6736-kserve-b86d7667b-smcbz" podStartSLOduration=1.532003969 podStartE2EDuration="1.532003969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:46:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:46:30.530857244 +0000 UTC m=+758.059888372" watchObservedRunningTime="2026-10-30 18:46:30.532003969 +0000 UTC m=+758.061035108" Oct 30 18:46:31.030814 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:31.030780 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-aa6736/distinct-w1-aa6736-kserve-b86d7667b-smcbz" Oct 30 18:46:31.274171 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:31.268696 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:31.401388 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:31.401353 6960 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-phklq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 18:46:31.401791 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:31.401410 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-phklq" podUID="67bc209e-b148-4464-912e-f1ba5f372c50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 18:46:32.404730 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:32.404681 6960 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-phklq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 18:46:32.405265 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:32.404786 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-phklq" podUID="67bc209e-b148-4464-912e-f1ba5f372c50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 18:46:32.449607 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:32.449574 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:33.401484 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:33.401452 6960 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-phklq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 18:46:33.401666 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:33.401504 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-phklq" podUID="67bc209e-b148-4464-912e-f1ba5f372c50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 18:46:33.408231 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:33.408206 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:34.332325 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:34.332275 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:34.402683 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:34.402085 6960 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-phklq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 18:46:34.402683 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:34.402138 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-phklq" podUID="67bc209e-b148-4464-912e-f1ba5f372c50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 18:46:35.223508 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:35.223477 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:35.991351 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:35.991323 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-phklq" Oct 30 18:46:35.992781 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:35.992760 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-phklq" Oct 30 18:46:35.996177 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:35.996159 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-phklq" Oct 30 18:46:36.120819 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:36.120793 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:37.061406 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:37.061373 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:38.040702 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:38.040647 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:39.066167 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:39.058472 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:39.903880 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:39.903834 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:40.031273 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:40.031242 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-aa6736/distinct-w1-aa6736-kserve-b86d7667b-smcbz" Oct 30 18:46:40.043750 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:40.043725 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-aa6736/distinct-w1-aa6736-kserve-b86d7667b-smcbz" Oct 30 18:46:40.611027 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:40.610995 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-aa6736/distinct-w1-aa6736-kserve-b86d7667b-smcbz" Oct 30 18:46:40.821614 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:40.821582 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:41.494273 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:41.494238 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-8a92cc/distinct-2-w2-8a92cc-kserve-6cc4b9c974-n7cd2"] Oct 30 18:46:41.498519 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:41.498499 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-2-w2-8a92cc-kserve-6cc4b9c974-n7cd2" Oct 30 18:46:41.514500 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:41.514479 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-8a92cc\"/\"distinct-2-w2-8a92cc-kserve-self-signed-certs\"" Oct 30 18:46:41.520168 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:41.520115 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-8a92cc/distinct-2-w2-8a92cc-kserve-6cc4b9c974-n7cd2"] Oct 30 18:46:41.572555 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:41.572522 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:41.586212 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:41.586182 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5fd8f33-8662-4f90-9817-e601f694bf1d-dshm\") pod \"distinct-2-w2-8a92cc-kserve-6cc4b9c974-n7cd2\" (UID: \"e5fd8f33-8662-4f90-9817-e601f694bf1d\") " pod="e2e-models-e2e-worker-w2-8a92cc/distinct-2-w2-8a92cc-kserve-6cc4b9c974-n7cd2" Oct 30 18:46:41.586394 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:41.586309 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5fd8f33-8662-4f90-9817-e601f694bf1d-model-cache\") pod \"distinct-2-w2-8a92cc-kserve-6cc4b9c974-n7cd2\" (UID: \"e5fd8f33-8662-4f90-9817-e601f694bf1d\") " pod="e2e-models-e2e-worker-w2-8a92cc/distinct-2-w2-8a92cc-kserve-6cc4b9c974-n7cd2" Oct 30 18:46:41.586456 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:41.586391 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5fd8f33-8662-4f90-9817-e601f694bf1d-tmp-dir\") pod \"distinct-2-w2-8a92cc-kserve-6cc4b9c974-n7cd2\" (UID: \"e5fd8f33-8662-4f90-9817-e601f694bf1d\") " pod="e2e-models-e2e-worker-w2-8a92cc/distinct-2-w2-8a92cc-kserve-6cc4b9c974-n7cd2" Oct 30 18:46:41.586456 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:41.586414 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5fd8f33-8662-4f90-9817-e601f694bf1d-home\") pod \"distinct-2-w2-8a92cc-kserve-6cc4b9c974-n7cd2\" (UID: \"e5fd8f33-8662-4f90-9817-e601f694bf1d\") " pod="e2e-models-e2e-worker-w2-8a92cc/distinct-2-w2-8a92cc-kserve-6cc4b9c974-n7cd2" Oct 30 18:46:41.586456 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:41.586442 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5fd8f33-8662-4f90-9817-e601f694bf1d-tls-certs\") pod \"distinct-2-w2-8a92cc-kserve-6cc4b9c974-n7cd2\" (UID: \"e5fd8f33-8662-4f90-9817-e601f694bf1d\") " pod="e2e-models-e2e-worker-w2-8a92cc/distinct-2-w2-8a92cc-kserve-6cc4b9c974-n7cd2" Oct 30 18:46:41.586616 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:41.586472 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbsd8\" (UniqueName: \"kubernetes.io/projected/e5fd8f33-8662-4f90-9817-e601f694bf1d-kube-api-access-pbsd8\") pod \"distinct-2-w2-8a92cc-kserve-6cc4b9c974-n7cd2\" (UID: \"e5fd8f33-8662-4f90-9817-e601f694bf1d\") " pod="e2e-models-e2e-worker-w2-8a92cc/distinct-2-w2-8a92cc-kserve-6cc4b9c974-n7cd2" Oct 30 18:46:41.688314 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:41.688280 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5fd8f33-8662-4f90-9817-e601f694bf1d-dshm\") pod \"distinct-2-w2-8a92cc-kserve-6cc4b9c974-n7cd2\" (UID: \"e5fd8f33-8662-4f90-9817-e601f694bf1d\") " pod="e2e-models-e2e-worker-w2-8a92cc/distinct-2-w2-8a92cc-kserve-6cc4b9c974-n7cd2" Oct 30 18:46:41.689084 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:41.689060 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5fd8f33-8662-4f90-9817-e601f694bf1d-model-cache\") pod \"distinct-2-w2-8a92cc-kserve-6cc4b9c974-n7cd2\" (UID: \"e5fd8f33-8662-4f90-9817-e601f694bf1d\") " pod="e2e-models-e2e-worker-w2-8a92cc/distinct-2-w2-8a92cc-kserve-6cc4b9c974-n7cd2" Oct 30 18:46:41.689332 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:41.689293 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5fd8f33-8662-4f90-9817-e601f694bf1d-tmp-dir\") pod \"distinct-2-w2-8a92cc-kserve-6cc4b9c974-n7cd2\" (UID: \"e5fd8f33-8662-4f90-9817-e601f694bf1d\") " pod="e2e-models-e2e-worker-w2-8a92cc/distinct-2-w2-8a92cc-kserve-6cc4b9c974-n7cd2" Oct 30 18:46:41.689457 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:41.689440 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5fd8f33-8662-4f90-9817-e601f694bf1d-home\") pod \"distinct-2-w2-8a92cc-kserve-6cc4b9c974-n7cd2\" (UID: \"e5fd8f33-8662-4f90-9817-e601f694bf1d\") " pod="e2e-models-e2e-worker-w2-8a92cc/distinct-2-w2-8a92cc-kserve-6cc4b9c974-n7cd2" Oct 30 18:46:41.689580 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:41.689567 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5fd8f33-8662-4f90-9817-e601f694bf1d-tls-certs\") pod \"distinct-2-w2-8a92cc-kserve-6cc4b9c974-n7cd2\" (UID: \"e5fd8f33-8662-4f90-9817-e601f694bf1d\") " pod="e2e-models-e2e-worker-w2-8a92cc/distinct-2-w2-8a92cc-kserve-6cc4b9c974-n7cd2" Oct 30 18:46:41.689678 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:41.689663 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbsd8\" (UniqueName: \"kubernetes.io/projected/e5fd8f33-8662-4f90-9817-e601f694bf1d-kube-api-access-pbsd8\") pod \"distinct-2-w2-8a92cc-kserve-6cc4b9c974-n7cd2\" (UID: \"e5fd8f33-8662-4f90-9817-e601f694bf1d\") " pod="e2e-models-e2e-worker-w2-8a92cc/distinct-2-w2-8a92cc-kserve-6cc4b9c974-n7cd2" Oct 30 18:46:41.689974 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:41.689842 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5fd8f33-8662-4f90-9817-e601f694bf1d-home\") pod \"distinct-2-w2-8a92cc-kserve-6cc4b9c974-n7cd2\" (UID: \"e5fd8f33-8662-4f90-9817-e601f694bf1d\") " pod="e2e-models-e2e-worker-w2-8a92cc/distinct-2-w2-8a92cc-kserve-6cc4b9c974-n7cd2" Oct 30 18:46:41.690061 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:41.689782 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5fd8f33-8662-4f90-9817-e601f694bf1d-tmp-dir\") pod \"distinct-2-w2-8a92cc-kserve-6cc4b9c974-n7cd2\" (UID: \"e5fd8f33-8662-4f90-9817-e601f694bf1d\") " pod="e2e-models-e2e-worker-w2-8a92cc/distinct-2-w2-8a92cc-kserve-6cc4b9c974-n7cd2" Oct 30 18:46:41.690061 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:41.689580 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5fd8f33-8662-4f90-9817-e601f694bf1d-model-cache\") pod \"distinct-2-w2-8a92cc-kserve-6cc4b9c974-n7cd2\" (UID: \"e5fd8f33-8662-4f90-9817-e601f694bf1d\") " pod="e2e-models-e2e-worker-w2-8a92cc/distinct-2-w2-8a92cc-kserve-6cc4b9c974-n7cd2" Oct 30 18:46:41.692350 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:41.692327 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5fd8f33-8662-4f90-9817-e601f694bf1d-dshm\") pod \"distinct-2-w2-8a92cc-kserve-6cc4b9c974-n7cd2\" (UID: \"e5fd8f33-8662-4f90-9817-e601f694bf1d\") " pod="e2e-models-e2e-worker-w2-8a92cc/distinct-2-w2-8a92cc-kserve-6cc4b9c974-n7cd2" Oct 30 18:46:41.692685 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:41.692669 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5fd8f33-8662-4f90-9817-e601f694bf1d-tls-certs\") pod \"distinct-2-w2-8a92cc-kserve-6cc4b9c974-n7cd2\" (UID: \"e5fd8f33-8662-4f90-9817-e601f694bf1d\") " pod="e2e-models-e2e-worker-w2-8a92cc/distinct-2-w2-8a92cc-kserve-6cc4b9c974-n7cd2" Oct 30 18:46:41.703300 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:41.703229 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbsd8\" (UniqueName: \"kubernetes.io/projected/e5fd8f33-8662-4f90-9817-e601f694bf1d-kube-api-access-pbsd8\") pod \"distinct-2-w2-8a92cc-kserve-6cc4b9c974-n7cd2\" (UID: \"e5fd8f33-8662-4f90-9817-e601f694bf1d\") " pod="e2e-models-e2e-worker-w2-8a92cc/distinct-2-w2-8a92cc-kserve-6cc4b9c974-n7cd2" Oct 30 18:46:41.813042 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:41.813008 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-2-w2-8a92cc-kserve-6cc4b9c974-n7cd2" Oct 30 18:46:41.898293 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:41.898091 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:42.098513 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:42.098489 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-8a92cc/distinct-2-w2-8a92cc-kserve-6cc4b9c974-n7cd2"] Oct 30 18:46:42.100825 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:46:42.100791 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode5fd8f33_8662_4f90_9817_e601f694bf1d.slice/crio-a2d176f3b8cc98642931f3eabb2f683a1e7b80f369c28aa1e9f3a02b3bb5fe82 WatchSource:0}: Error finding container a2d176f3b8cc98642931f3eabb2f683a1e7b80f369c28aa1e9f3a02b3bb5fe82: Status 404 returned error can't find the container with id a2d176f3b8cc98642931f3eabb2f683a1e7b80f369c28aa1e9f3a02b3bb5fe82 Oct 30 18:46:42.613264 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:42.613225 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-2-w2-8a92cc-kserve-6cc4b9c974-n7cd2" event={"ID":"e5fd8f33-8662-4f90-9817-e601f694bf1d","Type":"ContainerStarted","Data":"4fc2469d4f9c0f84d2cfad935cfca3034ebf064a33e3a578ffa150f56a040cf9"} Oct 30 18:46:42.613446 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:42.613274 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-2-w2-8a92cc-kserve-6cc4b9c974-n7cd2" event={"ID":"e5fd8f33-8662-4f90-9817-e601f694bf1d","Type":"ContainerStarted","Data":"a2d176f3b8cc98642931f3eabb2f683a1e7b80f369c28aa1e9f3a02b3bb5fe82"} Oct 30 18:46:42.632553 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:42.632501 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-2-w2-8a92cc-kserve-6cc4b9c974-n7cd2" podStartSLOduration=1.6324845959999998 podStartE2EDuration="1.632484596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:46:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:46:42.632190815 +0000 UTC m=+770.161221954" watchObservedRunningTime="2026-10-30 18:46:42.632484596 +0000 UTC m=+770.161515735" Oct 30 18:46:42.717617 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:42.717584 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:42.813833 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:42.813802 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-2-w2-8a92cc-kserve-6cc4b9c974-n7cd2" Oct 30 18:46:43.601076 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:43.601036 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:44.852199 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:44.852123 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:45.419249 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:45.419214 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:45.552168 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:45.546563 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:46.174636 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:46.174599 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:47.161434 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:47.161398 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:48.133926 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:48.133884 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:49.210099 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:49.210053 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:49.641584 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:49.641542 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:50.156779 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:50.156744 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:50.294614 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:50.294581 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:51.023196 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:51.023124 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:51.813529 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:51.813496 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-2-w2-8a92cc-kserve-6cc4b9c974-n7cd2" Oct 30 18:46:51.826284 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:51.826265 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-2-w2-8a92cc-kserve-6cc4b9c974-n7cd2" Oct 30 18:46:51.832667 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:51.832642 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:52.649564 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:52.649533 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:52.704794 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:52.704768 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-2-w2-8a92cc-kserve-6cc4b9c974-n7cd2" Oct 30 18:46:53.408870 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:53.408836 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:53.932276 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:46:53.932241 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 18:46:54.233359 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:54.233274 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:55.226750 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:55.226719 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:55.349389 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:55.349353 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:56.253079 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:56.253045 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:57.126695 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:57.126658 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:57.965076 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:57.965030 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:58.226384 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:58.226290 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:58.422633 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:58.422601 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:58.607815 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:46:58.607783 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:00.307018 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:00.306987 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:01.318798 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:01.318764 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:02.089943 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:02.089911 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:03.006870 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:03.006839 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:03.836892 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:03.836845 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:04.699286 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:04.699250 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:05.699785 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:05.699732 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:06.599234 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:06.599206 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:06.832541 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:06.832490 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:07.479217 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:07.479182 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:07.993762 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:07.993723 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:08.323319 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:08.323287 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:09.294052 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:09.294018 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:09.827944 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:09.827907 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-4thf5"] Oct 30 18:47:09.831867 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:09.831849 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-4thf5" Oct 30 18:47:09.837049 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:09.837029 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-aa6736-openshift-default-dockercfg-9lfh6\"" Oct 30 18:47:09.847272 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:09.847251 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-4thf5"] Oct 30 18:47:09.943262 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:09.943230 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c4b2f62-76f9-466c-b747-ee608a8bc6e3-workload-certs\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-4thf5\" (UID: \"1c4b2f62-76f9-466c-b747-ee608a8bc6e3\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-4thf5" Oct 30 18:47:09.943459 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:09.943353 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c4b2f62-76f9-466c-b747-ee608a8bc6e3-credential-socket\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-4thf5\" (UID: \"1c4b2f62-76f9-466c-b747-ee608a8bc6e3\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-4thf5" Oct 30 18:47:09.943459 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:09.943427 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c4b2f62-76f9-466c-b747-ee608a8bc6e3-workload-socket\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-4thf5\" (UID: \"1c4b2f62-76f9-466c-b747-ee608a8bc6e3\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-4thf5" Oct 30 18:47:09.943531 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:09.943508 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c4b2f62-76f9-466c-b747-ee608a8bc6e3-istio-token\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-4thf5\" (UID: \"1c4b2f62-76f9-466c-b747-ee608a8bc6e3\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-4thf5" Oct 30 18:47:09.943565 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:09.943546 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c4b2f62-76f9-466c-b747-ee608a8bc6e3-istio-envoy\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-4thf5\" (UID: \"1c4b2f62-76f9-466c-b747-ee608a8bc6e3\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-4thf5" Oct 30 18:47:09.943604 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:09.943578 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c4b2f62-76f9-466c-b747-ee608a8bc6e3-istiod-ca-cert\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-4thf5\" (UID: \"1c4b2f62-76f9-466c-b747-ee608a8bc6e3\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-4thf5" Oct 30 18:47:09.943673 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:09.943658 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c4b2f62-76f9-466c-b747-ee608a8bc6e3-istio-podinfo\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-4thf5\" (UID: \"1c4b2f62-76f9-466c-b747-ee608a8bc6e3\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-4thf5" Oct 30 18:47:09.943708 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:09.943700 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c4b2f62-76f9-466c-b747-ee608a8bc6e3-istio-data\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-4thf5\" (UID: \"1c4b2f62-76f9-466c-b747-ee608a8bc6e3\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-4thf5" Oct 30 18:47:09.943742 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:09.943725 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt4dm\" (UniqueName: \"kubernetes.io/projected/1c4b2f62-76f9-466c-b747-ee608a8bc6e3-kube-api-access-nt4dm\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-4thf5\" (UID: \"1c4b2f62-76f9-466c-b747-ee608a8bc6e3\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-4thf5" Oct 30 18:47:10.044951 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:10.044919 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c4b2f62-76f9-466c-b747-ee608a8bc6e3-istio-token\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-4thf5\" (UID: \"1c4b2f62-76f9-466c-b747-ee608a8bc6e3\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-4thf5" Oct 30 18:47:10.044951 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:10.044954 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c4b2f62-76f9-466c-b747-ee608a8bc6e3-istio-envoy\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-4thf5\" (UID: \"1c4b2f62-76f9-466c-b747-ee608a8bc6e3\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-4thf5" Oct 30 18:47:10.045219 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:10.045002 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c4b2f62-76f9-466c-b747-ee608a8bc6e3-istiod-ca-cert\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-4thf5\" (UID: \"1c4b2f62-76f9-466c-b747-ee608a8bc6e3\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-4thf5" Oct 30 18:47:10.045219 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:10.045033 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c4b2f62-76f9-466c-b747-ee608a8bc6e3-istio-podinfo\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-4thf5\" (UID: \"1c4b2f62-76f9-466c-b747-ee608a8bc6e3\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-4thf5" Oct 30 18:47:10.045219 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:10.045061 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c4b2f62-76f9-466c-b747-ee608a8bc6e3-istio-data\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-4thf5\" (UID: \"1c4b2f62-76f9-466c-b747-ee608a8bc6e3\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-4thf5" Oct 30 18:47:10.045219 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:10.045085 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt4dm\" (UniqueName: \"kubernetes.io/projected/1c4b2f62-76f9-466c-b747-ee608a8bc6e3-kube-api-access-nt4dm\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-4thf5\" (UID: \"1c4b2f62-76f9-466c-b747-ee608a8bc6e3\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-4thf5" Oct 30 18:47:10.045219 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:10.045125 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c4b2f62-76f9-466c-b747-ee608a8bc6e3-workload-certs\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-4thf5\" (UID: \"1c4b2f62-76f9-466c-b747-ee608a8bc6e3\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-4thf5" Oct 30 18:47:10.045498 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:10.045252 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c4b2f62-76f9-466c-b747-ee608a8bc6e3-credential-socket\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-4thf5\" (UID: \"1c4b2f62-76f9-466c-b747-ee608a8bc6e3\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-4thf5" Oct 30 18:47:10.045498 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:10.045343 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c4b2f62-76f9-466c-b747-ee608a8bc6e3-workload-socket\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-4thf5\" (UID: \"1c4b2f62-76f9-466c-b747-ee608a8bc6e3\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-4thf5" Oct 30 18:47:10.045603 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:10.045505 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c4b2f62-76f9-466c-b747-ee608a8bc6e3-istio-data\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-4thf5\" (UID: \"1c4b2f62-76f9-466c-b747-ee608a8bc6e3\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-4thf5" Oct 30 18:47:10.045657 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:10.045606 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c4b2f62-76f9-466c-b747-ee608a8bc6e3-workload-certs\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-4thf5\" (UID: \"1c4b2f62-76f9-466c-b747-ee608a8bc6e3\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-4thf5" Oct 30 18:47:10.045713 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:10.045685 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c4b2f62-76f9-466c-b747-ee608a8bc6e3-credential-socket\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-4thf5\" (UID: \"1c4b2f62-76f9-466c-b747-ee608a8bc6e3\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-4thf5" Oct 30 18:47:10.045713 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:10.045698 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c4b2f62-76f9-466c-b747-ee608a8bc6e3-workload-socket\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-4thf5\" (UID: \"1c4b2f62-76f9-466c-b747-ee608a8bc6e3\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-4thf5" Oct 30 18:47:10.046055 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:10.046031 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c4b2f62-76f9-466c-b747-ee608a8bc6e3-istiod-ca-cert\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-4thf5\" (UID: \"1c4b2f62-76f9-466c-b747-ee608a8bc6e3\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-4thf5" Oct 30 18:47:10.047722 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:10.047695 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c4b2f62-76f9-466c-b747-ee608a8bc6e3-istio-envoy\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-4thf5\" (UID: \"1c4b2f62-76f9-466c-b747-ee608a8bc6e3\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-4thf5" Oct 30 18:47:10.047976 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:10.047786 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c4b2f62-76f9-466c-b747-ee608a8bc6e3-istio-podinfo\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-4thf5\" (UID: \"1c4b2f62-76f9-466c-b747-ee608a8bc6e3\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-4thf5" Oct 30 18:47:10.062346 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:10.062326 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt4dm\" (UniqueName: \"kubernetes.io/projected/1c4b2f62-76f9-466c-b747-ee608a8bc6e3-kube-api-access-nt4dm\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-4thf5\" (UID: \"1c4b2f62-76f9-466c-b747-ee608a8bc6e3\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-4thf5" Oct 30 18:47:10.062733 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:10.062713 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c4b2f62-76f9-466c-b747-ee608a8bc6e3-istio-token\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-4thf5\" (UID: \"1c4b2f62-76f9-466c-b747-ee608a8bc6e3\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-4thf5" Oct 30 18:47:10.131049 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:10.130985 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:10.145878 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:10.145857 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-4thf5" Oct 30 18:47:10.438183 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:10.438138 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-4thf5"] Oct 30 18:47:10.438934 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:47:10.438907 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c4b2f62_76f9_466c_b747_ee608a8bc6e3.slice/crio-cba0185495e59ce0477adc09bdd476b5980bede1b2bb24e6a114be902b3024e0 WatchSource:0}: Error finding container cba0185495e59ce0477adc09bdd476b5980bede1b2bb24e6a114be902b3024e0: Status 404 returned error can't find the container with id cba0185495e59ce0477adc09bdd476b5980bede1b2bb24e6a114be902b3024e0 Oct 30 18:47:10.441177 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:10.441123 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:47:10.441244 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:10.441205 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:47:10.441244 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:10.441235 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:47:10.934241 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:10.934207 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:10.982475 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:10.982442 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-4thf5" event={"ID":"1c4b2f62-76f9-466c-b747-ee608a8bc6e3","Type":"ContainerStarted","Data":"94a79476ef6b20859e2524b21ad96a7da3bfee23909321b8534ebbce294aacc2"} Oct 30 18:47:10.982653 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:10.982494 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-4thf5" event={"ID":"1c4b2f62-76f9-466c-b747-ee608a8bc6e3","Type":"ContainerStarted","Data":"cba0185495e59ce0477adc09bdd476b5980bede1b2bb24e6a114be902b3024e0"} Oct 30 18:47:11.025762 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:11.025646 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-4thf5" podStartSLOduration=2.02562677 podStartE2EDuration="2.02562677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:47:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:47:11.024360613 +0000 UTC m=+798.553391752" watchObservedRunningTime="2026-10-30 18:47:11.02562677 +0000 UTC m=+798.554657909" Oct 30 18:47:11.106746 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:11.106710 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:11.149566 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:11.149524 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-4thf5" Oct 30 18:47:11.155169 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:11.153297 6960 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-4thf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 18:47:11.155169 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:11.153373 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-4thf5" podUID="1c4b2f62-76f9-466c-b747-ee608a8bc6e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 18:47:12.042798 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:12.042769 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:12.146912 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:12.146878 6960 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-4thf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 18:47:12.147078 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:12.146931 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-4thf5" podUID="1c4b2f62-76f9-466c-b747-ee608a8bc6e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 18:47:12.948810 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:12.948778 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:13.146706 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:13.146675 6960 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-4thf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 18:47:13.147089 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:13.146724 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-4thf5" podUID="1c4b2f62-76f9-466c-b747-ee608a8bc6e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 18:47:13.925123 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:13.925084 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:14.152165 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:14.147562 6960 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-4thf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 18:47:14.152165 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:14.147624 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-4thf5" podUID="1c4b2f62-76f9-466c-b747-ee608a8bc6e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 18:47:14.851170 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:14.844835 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-5m6bs"] Oct 30 18:47:14.859213 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:14.855436 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-5m6bs" Oct 30 18:47:14.867166 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:14.866683 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-5m6bs"] Oct 30 18:47:14.985171 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:14.974453 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0a610f7-6daf-48de-865d-a0788c9d036a-istio-podinfo\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-5m6bs\" (UID: \"f0a610f7-6daf-48de-865d-a0788c9d036a\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-5m6bs" Oct 30 18:47:14.985171 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:14.974541 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0a610f7-6daf-48de-865d-a0788c9d036a-workload-certs\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-5m6bs\" (UID: \"f0a610f7-6daf-48de-865d-a0788c9d036a\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-5m6bs" Oct 30 18:47:14.985171 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:14.974578 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0a610f7-6daf-48de-865d-a0788c9d036a-istio-data\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-5m6bs\" (UID: \"f0a610f7-6daf-48de-865d-a0788c9d036a\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-5m6bs" Oct 30 18:47:14.985171 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:14.974619 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0a610f7-6daf-48de-865d-a0788c9d036a-workload-socket\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-5m6bs\" (UID: \"f0a610f7-6daf-48de-865d-a0788c9d036a\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-5m6bs" Oct 30 18:47:14.985171 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:14.974645 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0a610f7-6daf-48de-865d-a0788c9d036a-istio-envoy\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-5m6bs\" (UID: \"f0a610f7-6daf-48de-865d-a0788c9d036a\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-5m6bs" Oct 30 18:47:14.985171 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:14.974700 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0a610f7-6daf-48de-865d-a0788c9d036a-istio-token\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-5m6bs\" (UID: \"f0a610f7-6daf-48de-865d-a0788c9d036a\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-5m6bs" Oct 30 18:47:14.985171 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:14.974724 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0a610f7-6daf-48de-865d-a0788c9d036a-istiod-ca-cert\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-5m6bs\" (UID: \"f0a610f7-6daf-48de-865d-a0788c9d036a\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-5m6bs" Oct 30 18:47:14.985171 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:14.974776 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl2zr\" (UniqueName: \"kubernetes.io/projected/f0a610f7-6daf-48de-865d-a0788c9d036a-kube-api-access-rl2zr\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-5m6bs\" (UID: \"f0a610f7-6daf-48de-865d-a0788c9d036a\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-5m6bs" Oct 30 18:47:14.985171 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:14.974814 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0a610f7-6daf-48de-865d-a0788c9d036a-credential-socket\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-5m6bs\" (UID: \"f0a610f7-6daf-48de-865d-a0788c9d036a\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-5m6bs" Oct 30 18:47:15.108252 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:15.101707 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0a610f7-6daf-48de-865d-a0788c9d036a-istio-token\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-5m6bs\" (UID: \"f0a610f7-6daf-48de-865d-a0788c9d036a\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-5m6bs" Oct 30 18:47:15.108252 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:15.101755 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0a610f7-6daf-48de-865d-a0788c9d036a-istiod-ca-cert\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-5m6bs\" (UID: \"f0a610f7-6daf-48de-865d-a0788c9d036a\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-5m6bs" Oct 30 18:47:15.108252 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:15.101806 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl2zr\" (UniqueName: \"kubernetes.io/projected/f0a610f7-6daf-48de-865d-a0788c9d036a-kube-api-access-rl2zr\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-5m6bs\" (UID: \"f0a610f7-6daf-48de-865d-a0788c9d036a\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-5m6bs" Oct 30 18:47:15.108252 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:15.101842 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0a610f7-6daf-48de-865d-a0788c9d036a-credential-socket\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-5m6bs\" (UID: \"f0a610f7-6daf-48de-865d-a0788c9d036a\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-5m6bs" Oct 30 18:47:15.108252 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:15.101998 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0a610f7-6daf-48de-865d-a0788c9d036a-istio-podinfo\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-5m6bs\" (UID: \"f0a610f7-6daf-48de-865d-a0788c9d036a\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-5m6bs" Oct 30 18:47:15.108252 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:15.102060 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0a610f7-6daf-48de-865d-a0788c9d036a-workload-certs\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-5m6bs\" (UID: \"f0a610f7-6daf-48de-865d-a0788c9d036a\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-5m6bs" Oct 30 18:47:15.108252 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:15.102089 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0a610f7-6daf-48de-865d-a0788c9d036a-istio-data\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-5m6bs\" (UID: \"f0a610f7-6daf-48de-865d-a0788c9d036a\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-5m6bs" Oct 30 18:47:15.108252 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:15.102127 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0a610f7-6daf-48de-865d-a0788c9d036a-workload-socket\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-5m6bs\" (UID: \"f0a610f7-6daf-48de-865d-a0788c9d036a\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-5m6bs" Oct 30 18:47:15.108252 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:15.102174 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0a610f7-6daf-48de-865d-a0788c9d036a-istio-envoy\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-5m6bs\" (UID: \"f0a610f7-6daf-48de-865d-a0788c9d036a\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-5m6bs" Oct 30 18:47:15.120915 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:15.120881 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0a610f7-6daf-48de-865d-a0788c9d036a-credential-socket\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-5m6bs\" (UID: \"f0a610f7-6daf-48de-865d-a0788c9d036a\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-5m6bs" Oct 30 18:47:15.142203 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:15.122052 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0a610f7-6daf-48de-865d-a0788c9d036a-istiod-ca-cert\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-5m6bs\" (UID: \"f0a610f7-6daf-48de-865d-a0788c9d036a\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-5m6bs" Oct 30 18:47:15.142431 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:15.123690 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0a610f7-6daf-48de-865d-a0788c9d036a-workload-certs\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-5m6bs\" (UID: \"f0a610f7-6daf-48de-865d-a0788c9d036a\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-5m6bs" Oct 30 18:47:15.142575 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:15.140519 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0a610f7-6daf-48de-865d-a0788c9d036a-istio-data\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-5m6bs\" (UID: \"f0a610f7-6daf-48de-865d-a0788c9d036a\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-5m6bs" Oct 30 18:47:15.142701 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:15.140745 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0a610f7-6daf-48de-865d-a0788c9d036a-workload-socket\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-5m6bs\" (UID: \"f0a610f7-6daf-48de-865d-a0788c9d036a\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-5m6bs" Oct 30 18:47:15.142846 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:15.141537 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0a610f7-6daf-48de-865d-a0788c9d036a-istio-podinfo\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-5m6bs\" (UID: \"f0a610f7-6daf-48de-865d-a0788c9d036a\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-5m6bs" Oct 30 18:47:15.148892 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:15.148500 6960 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-4thf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 18:47:15.148892 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:15.148549 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-4thf5" podUID="1c4b2f62-76f9-466c-b747-ee608a8bc6e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 18:47:15.193169 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:15.190650 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0a610f7-6daf-48de-865d-a0788c9d036a-istio-token\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-5m6bs\" (UID: \"f0a610f7-6daf-48de-865d-a0788c9d036a\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-5m6bs" Oct 30 18:47:15.222170 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:15.216513 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0a610f7-6daf-48de-865d-a0788c9d036a-istio-envoy\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-5m6bs\" (UID: \"f0a610f7-6daf-48de-865d-a0788c9d036a\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-5m6bs" Oct 30 18:47:15.258169 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:15.257373 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl2zr\" (UniqueName: \"kubernetes.io/projected/f0a610f7-6daf-48de-865d-a0788c9d036a-kube-api-access-rl2zr\") pod \"e2e-worker-w4-da70a5-openshift-default-bb4f4b65-5m6bs\" (UID: \"f0a610f7-6daf-48de-865d-a0788c9d036a\") " pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-5m6bs" Oct 30 18:47:15.439050 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:15.438733 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-5m6bs" Oct 30 18:47:15.548799 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:15.547228 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/simulator-w2-8a92cc-kserve-fb77d84bc-nqf9q" podUID="ce70492e-c727-4dc1-b2ee-bf278d29a45b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:47:15.628296 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:15.627237 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-aa6736/distinct-w1-aa6736-kserve-b86d7667b-smcbz" podUID="58c23db7-11cb-4ce2-a67e-fe698e89c71d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:47:15.671344 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:15.670279 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7a8fa2/premium-w3-7a8fa2-kserve-6b6c5f6fb8-n5wtq" podUID="f51fd746-cd85-4608-96a6-f54afa0ec34d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 30 18:47:15.700765 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:15.699678 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-2-w2-8a92cc-kserve-6cc4b9c974-n7cd2" podUID="e5fd8f33-8662-4f90-9817-e601f694bf1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 18:47:15.792394 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:15.791295 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-aa6736/premium-w1-aa6736-kserve-6f986b66f7-nkfrg" podUID="84ca40c5-7033-48c4-bcec-e9ac7c034a4e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:47:16.069278 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:16.068286 6960 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-d58l4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:47:16.069278 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:16.068355 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d58l4" podUID="42b9fddf-ca5b-4181-8819-c32996d6ff0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:47:16.103170 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:16.094249 6960 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-pb8bs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:47:16.111339 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:16.111296 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pb8bs" podUID="830b7d8c-c70c-4242-b81a-8be190df4aa1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:47:16.414866 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:16.413489 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:16.439167 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:16.416548 6960 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-4thf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 18:47:16.439167 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:16.416597 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-4thf5" podUID="1c4b2f62-76f9-466c-b747-ee608a8bc6e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 18:47:16.487722 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:16.485976 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-5m6bs"] Oct 30 18:47:16.542561 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:16.542437 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:47:16.542894 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:16.542867 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:47:16.543359 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:16.543324 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:47:16.723218 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:16.698161 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7a8fa2/premium-w3-7a8fa2-kserve-6b6c5f6fb8-n5wtq" podUID="f51fd746-cd85-4608-96a6-f54afa0ec34d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:47:16.724013 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:16.706371 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-2-w2-8a92cc-kserve-6cc4b9c974-n7cd2" podUID="e5fd8f33-8662-4f90-9817-e601f694bf1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:47:16.727219 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:16.724633 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7a8fa2/premium-w3-7a8fa2-kserve-6b6c5f6fb8-n5wtq" Oct 30 18:47:16.735175 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:16.727435 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-2-w2-8a92cc-kserve-6cc4b9c974-n7cd2" Oct 30 18:47:16.881278 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:16.842663 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-aa6736/premium-w1-aa6736-kserve-6f986b66f7-nkfrg" podUID="84ca40c5-7033-48c4-bcec-e9ac7c034a4e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:47:16.909612 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:16.908492 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-aa6736/premium-w1-aa6736-kserve-6f986b66f7-nkfrg" Oct 30 18:47:17.148381 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:17.147455 6960 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-4thf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 18:47:17.148381 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:17.147507 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-4thf5" podUID="1c4b2f62-76f9-466c-b747-ee608a8bc6e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 18:47:17.203876 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:17.203846 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-hj7n6"] Oct 30 18:47:17.531355 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:17.531276 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7a8fa2/premium-w3-7a8fa2-kserve-6b6c5f6fb8-n5wtq" Oct 30 18:47:17.539257 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:17.539231 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-hj7n6"] Oct 30 18:47:17.539541 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:17.532068 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-hj7n6" Oct 30 18:47:17.539759 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:17.539744 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-2-w2-8a92cc-kserve-6cc4b9c974-n7cd2" Oct 30 18:47:17.541642 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:17.541613 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-aa6736/premium-w1-aa6736-kserve-6f986b66f7-nkfrg" Oct 30 18:47:17.552495 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:17.551427 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-5m6bs" event={"ID":"f0a610f7-6daf-48de-865d-a0788c9d036a","Type":"ContainerStarted","Data":"b93ffb29a8381e5b78b67b8c947c389bbfcf45da2e9590695e0c27f022c09885"} Oct 30 18:47:17.552495 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:17.551477 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-5m6bs" event={"ID":"f0a610f7-6daf-48de-865d-a0788c9d036a","Type":"ContainerStarted","Data":"56dfee22b633efd8b22f4910a79a7b8a22ab7525881023d3889bb6ce3525160a"} Oct 30 18:47:17.560414 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:17.560391 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-8a92cc-openshift-default-dockercfg-vj746\"" Oct 30 18:47:17.722029 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:17.721970 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-5m6bs" podStartSLOduration=3.721950164 podStartE2EDuration="3.721950164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:47:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:47:17.721135833 +0000 UTC m=+805.250166984" watchObservedRunningTime="2026-10-30 18:47:17.721950164 +0000 UTC m=+805.250981304" Oct 30 18:47:17.732790 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:17.732732 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71c3b467-e87a-40eb-ad20-7175b0db9e49-istiod-ca-cert\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-hj7n6\" (UID: \"71c3b467-e87a-40eb-ad20-7175b0db9e49\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-hj7n6" Oct 30 18:47:17.733659 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:17.733621 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71c3b467-e87a-40eb-ad20-7175b0db9e49-workload-certs\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-hj7n6\" (UID: \"71c3b467-e87a-40eb-ad20-7175b0db9e49\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-hj7n6" Oct 30 18:47:17.734022 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:17.733990 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71c3b467-e87a-40eb-ad20-7175b0db9e49-istio-podinfo\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-hj7n6\" (UID: \"71c3b467-e87a-40eb-ad20-7175b0db9e49\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-hj7n6" Oct 30 18:47:17.734257 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:17.734226 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71c3b467-e87a-40eb-ad20-7175b0db9e49-workload-socket\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-hj7n6\" (UID: \"71c3b467-e87a-40eb-ad20-7175b0db9e49\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-hj7n6" Oct 30 18:47:17.735102 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:17.735080 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71c3b467-e87a-40eb-ad20-7175b0db9e49-credential-socket\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-hj7n6\" (UID: \"71c3b467-e87a-40eb-ad20-7175b0db9e49\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-hj7n6" Oct 30 18:47:17.735398 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:17.735378 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44s46\" (UniqueName: \"kubernetes.io/projected/71c3b467-e87a-40eb-ad20-7175b0db9e49-kube-api-access-44s46\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-hj7n6\" (UID: \"71c3b467-e87a-40eb-ad20-7175b0db9e49\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-hj7n6" Oct 30 18:47:17.735630 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:17.735588 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71c3b467-e87a-40eb-ad20-7175b0db9e49-istio-data\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-hj7n6\" (UID: \"71c3b467-e87a-40eb-ad20-7175b0db9e49\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-hj7n6" Oct 30 18:47:17.736189 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:17.735854 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71c3b467-e87a-40eb-ad20-7175b0db9e49-istio-token\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-hj7n6\" (UID: \"71c3b467-e87a-40eb-ad20-7175b0db9e49\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-hj7n6" Oct 30 18:47:17.736189 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:17.735890 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71c3b467-e87a-40eb-ad20-7175b0db9e49-istio-envoy\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-hj7n6\" (UID: \"71c3b467-e87a-40eb-ad20-7175b0db9e49\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-hj7n6" Oct 30 18:47:17.838754 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:17.837461 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71c3b467-e87a-40eb-ad20-7175b0db9e49-workload-certs\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-hj7n6\" (UID: \"71c3b467-e87a-40eb-ad20-7175b0db9e49\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-hj7n6" Oct 30 18:47:17.838754 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:17.837548 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71c3b467-e87a-40eb-ad20-7175b0db9e49-istio-podinfo\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-hj7n6\" (UID: \"71c3b467-e87a-40eb-ad20-7175b0db9e49\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-hj7n6" Oct 30 18:47:17.838754 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:17.837599 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71c3b467-e87a-40eb-ad20-7175b0db9e49-workload-socket\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-hj7n6\" (UID: \"71c3b467-e87a-40eb-ad20-7175b0db9e49\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-hj7n6" Oct 30 18:47:17.838754 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:17.837633 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71c3b467-e87a-40eb-ad20-7175b0db9e49-credential-socket\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-hj7n6\" (UID: \"71c3b467-e87a-40eb-ad20-7175b0db9e49\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-hj7n6" Oct 30 18:47:17.838754 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:17.837690 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44s46\" (UniqueName: \"kubernetes.io/projected/71c3b467-e87a-40eb-ad20-7175b0db9e49-kube-api-access-44s46\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-hj7n6\" (UID: \"71c3b467-e87a-40eb-ad20-7175b0db9e49\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-hj7n6" Oct 30 18:47:17.838754 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:17.837716 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71c3b467-e87a-40eb-ad20-7175b0db9e49-istio-data\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-hj7n6\" (UID: \"71c3b467-e87a-40eb-ad20-7175b0db9e49\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-hj7n6" Oct 30 18:47:17.838754 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:17.837756 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71c3b467-e87a-40eb-ad20-7175b0db9e49-istio-token\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-hj7n6\" (UID: \"71c3b467-e87a-40eb-ad20-7175b0db9e49\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-hj7n6" Oct 30 18:47:17.838754 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:17.837783 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71c3b467-e87a-40eb-ad20-7175b0db9e49-istio-envoy\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-hj7n6\" (UID: \"71c3b467-e87a-40eb-ad20-7175b0db9e49\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-hj7n6" Oct 30 18:47:17.838754 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:17.837832 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71c3b467-e87a-40eb-ad20-7175b0db9e49-istiod-ca-cert\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-hj7n6\" (UID: \"71c3b467-e87a-40eb-ad20-7175b0db9e49\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-hj7n6" Oct 30 18:47:17.841255 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:17.839904 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71c3b467-e87a-40eb-ad20-7175b0db9e49-istiod-ca-cert\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-hj7n6\" (UID: \"71c3b467-e87a-40eb-ad20-7175b0db9e49\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-hj7n6" Oct 30 18:47:17.841255 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:17.840279 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71c3b467-e87a-40eb-ad20-7175b0db9e49-credential-socket\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-hj7n6\" (UID: \"71c3b467-e87a-40eb-ad20-7175b0db9e49\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-hj7n6" Oct 30 18:47:17.841255 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:17.840757 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71c3b467-e87a-40eb-ad20-7175b0db9e49-istio-data\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-hj7n6\" (UID: \"71c3b467-e87a-40eb-ad20-7175b0db9e49\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-hj7n6" Oct 30 18:47:17.841255 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:17.841225 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71c3b467-e87a-40eb-ad20-7175b0db9e49-workload-socket\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-hj7n6\" (UID: \"71c3b467-e87a-40eb-ad20-7175b0db9e49\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-hj7n6" Oct 30 18:47:17.841443 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:17.841352 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71c3b467-e87a-40eb-ad20-7175b0db9e49-workload-certs\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-hj7n6\" (UID: \"71c3b467-e87a-40eb-ad20-7175b0db9e49\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-hj7n6" Oct 30 18:47:17.844772 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:17.844750 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71c3b467-e87a-40eb-ad20-7175b0db9e49-istio-podinfo\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-hj7n6\" (UID: \"71c3b467-e87a-40eb-ad20-7175b0db9e49\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-hj7n6" Oct 30 18:47:17.847484 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:17.847409 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71c3b467-e87a-40eb-ad20-7175b0db9e49-istio-envoy\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-hj7n6\" (UID: \"71c3b467-e87a-40eb-ad20-7175b0db9e49\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-hj7n6" Oct 30 18:47:17.857962 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:17.854726 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71c3b467-e87a-40eb-ad20-7175b0db9e49-istio-token\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-hj7n6\" (UID: \"71c3b467-e87a-40eb-ad20-7175b0db9e49\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-hj7n6" Oct 30 18:47:17.857962 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:17.857090 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44s46\" (UniqueName: \"kubernetes.io/projected/71c3b467-e87a-40eb-ad20-7175b0db9e49-kube-api-access-44s46\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-hj7n6\" (UID: \"71c3b467-e87a-40eb-ad20-7175b0db9e49\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-hj7n6" Oct 30 18:47:17.898218 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:17.891296 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-hj7n6" Oct 30 18:47:18.046338 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:18.046203 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:18.147175 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:18.147050 6960 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-4thf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 18:47:18.147175 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:18.147095 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-4thf5" podUID="1c4b2f62-76f9-466c-b747-ee608a8bc6e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 18:47:18.228197 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:18.226595 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-hj7n6"] Oct 30 18:47:18.239468 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:47:18.239421 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71c3b467_e87a_40eb_ad20_7175b0db9e49.slice/crio-6f3625567372ed03860620b36ebee3f07d1143f5e7f077c1bc918d628bd71df6 WatchSource:0}: Error finding container 6f3625567372ed03860620b36ebee3f07d1143f5e7f077c1bc918d628bd71df6: Status 404 returned error can't find the container with id 6f3625567372ed03860620b36ebee3f07d1143f5e7f077c1bc918d628bd71df6 Oct 30 18:47:18.242173 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:18.242039 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:47:18.245002 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:18.242690 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:47:18.245002 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:18.242760 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:47:18.411230 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:18.411084 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:18.441060 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:18.441022 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-5m6bs" Oct 30 18:47:18.442331 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:18.442296 6960 patch_prober.go:29] interesting pod/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-5m6bs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 18:47:18.442480 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:18.442357 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-5m6bs" podUID="f0a610f7-6daf-48de-865d-a0788c9d036a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 18:47:18.526030 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:18.525996 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:18.574182 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:18.570903 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-hj7n6" event={"ID":"71c3b467-e87a-40eb-ad20-7175b0db9e49","Type":"ContainerStarted","Data":"38f11f8f890154aab95d5bb526fa1386b6f2b41ddd9140624c1c28a4afc0d9c0"} Oct 30 18:47:18.574182 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:18.570959 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-hj7n6" event={"ID":"71c3b467-e87a-40eb-ad20-7175b0db9e49","Type":"ContainerStarted","Data":"6f3625567372ed03860620b36ebee3f07d1143f5e7f077c1bc918d628bd71df6"} Oct 30 18:47:18.910588 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:18.910552 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-hj7n6" Oct 30 18:47:19.283849 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:19.283609 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:19.439651 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:19.439627 6960 patch_prober.go:29] interesting pod/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-5m6bs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 18:47:19.439863 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:19.439844 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-5m6bs" podUID="f0a610f7-6daf-48de-865d-a0788c9d036a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 18:47:19.917630 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:19.904264 6960 patch_prober.go:29] interesting pod/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-hj7n6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 18:47:19.917630 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:19.904325 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-hj7n6" podUID="71c3b467-e87a-40eb-ad20-7175b0db9e49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 18:47:20.205364 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:20.204273 6960 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-4thf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:47:20.205364 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:20.204358 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-4thf5" podUID="1c4b2f62-76f9-466c-b747-ee608a8bc6e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:47:20.440607 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:20.440568 6960 patch_prober.go:29] interesting pod/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-5m6bs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 18:47:20.440814 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:20.440646 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-5m6bs" podUID="f0a610f7-6daf-48de-865d-a0788c9d036a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 18:47:20.563433 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:20.563340 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:20.630835 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:20.630746 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-4thf5" Oct 30 18:47:20.631410 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:20.631389 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-4thf5" Oct 30 18:47:20.633288 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:20.633259 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-4thf5" Oct 30 18:47:20.670988 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:20.670926 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-hj7n6" podStartSLOduration=3.670905788 podStartE2EDuration="3.670905788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:47:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:47:18.617792295 +0000 UTC m=+806.146823434" watchObservedRunningTime="2026-10-30 18:47:20.670905788 +0000 UTC m=+808.199936924" Oct 30 18:47:20.895623 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:20.895537 6960 patch_prober.go:29] interesting pod/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-hj7n6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 18:47:20.895913 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:20.895872 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-hj7n6" podUID="71c3b467-e87a-40eb-ad20-7175b0db9e49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 18:47:21.440281 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:21.440253 6960 patch_prober.go:29] interesting pod/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-5m6bs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 18:47:21.440710 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:21.440686 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-5m6bs" podUID="f0a610f7-6daf-48de-865d-a0788c9d036a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 18:47:21.716265 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:21.716181 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:21.894169 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:21.893282 6960 patch_prober.go:29] interesting pod/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-hj7n6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 18:47:21.894169 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:21.893346 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-hj7n6" podUID="71c3b467-e87a-40eb-ad20-7175b0db9e49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 18:47:22.440360 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:22.440324 6960 patch_prober.go:29] interesting pod/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-5m6bs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 18:47:22.440743 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:22.440378 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-5m6bs" podUID="f0a610f7-6daf-48de-865d-a0788c9d036a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 18:47:22.897168 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:22.896579 6960 patch_prober.go:29] interesting pod/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-hj7n6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 18:47:22.897168 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:22.896641 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-hj7n6" podUID="71c3b467-e87a-40eb-ad20-7175b0db9e49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 18:47:22.915895 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:22.915864 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:23.439812 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:23.439781 6960 patch_prober.go:29] interesting pod/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-5m6bs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 18:47:23.439968 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:23.439847 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-5m6bs" podUID="f0a610f7-6daf-48de-865d-a0788c9d036a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 18:47:23.892695 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:23.892656 6960 patch_prober.go:29] interesting pod/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-hj7n6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 18:47:23.893102 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:23.892725 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-hj7n6" podUID="71c3b467-e87a-40eb-ad20-7175b0db9e49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 18:47:23.935681 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:47:23.935648 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 18:47:23.977167 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:23.976676 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:24.440098 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:24.440063 6960 patch_prober.go:29] interesting pod/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-5m6bs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 18:47:24.440291 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:24.440128 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-5m6bs" podUID="f0a610f7-6daf-48de-865d-a0788c9d036a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 18:47:24.532504 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:24.532470 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:24.826851 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:24.826808 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:24.898176 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:24.893523 6960 patch_prober.go:29] interesting pod/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-hj7n6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 18:47:24.898176 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:24.893581 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-hj7n6" podUID="71c3b467-e87a-40eb-ad20-7175b0db9e49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 18:47:25.444926 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:25.444895 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-5m6bs" Oct 30 18:47:25.445224 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:25.445205 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-5m6bs" Oct 30 18:47:25.446312 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:25.446292 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-5m6bs" Oct 30 18:47:25.952709 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:25.952678 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:26.155276 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:26.155235 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-7a8fa2/distinct-2-w3-7a8fa2-kserve-677bb44d8d-fg55k"] Oct 30 18:47:26.159712 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:26.159690 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-hj7n6" Oct 30 18:47:26.159860 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:26.159824 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-2-w3-7a8fa2-kserve-677bb44d8d-fg55k" Oct 30 18:47:26.160022 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:26.160001 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-hj7n6" Oct 30 18:47:26.160868 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:26.160845 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-hj7n6" Oct 30 18:47:26.174514 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:26.174486 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-7a8fa2\"/\"distinct-2-w3-7a8fa2-kserve-self-signed-certs\"" Oct 30 18:47:26.183726 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:26.183698 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-7a8fa2/distinct-2-w3-7a8fa2-kserve-677bb44d8d-fg55k"] Oct 30 18:47:26.233923 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:26.233803 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:26.285938 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:26.285897 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8701242-b59c-43b1-9c0a-011635edd419-dshm\") pod \"distinct-2-w3-7a8fa2-kserve-677bb44d8d-fg55k\" (UID: \"e8701242-b59c-43b1-9c0a-011635edd419\") " pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-2-w3-7a8fa2-kserve-677bb44d8d-fg55k" Oct 30 18:47:26.286209 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:26.286189 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8701242-b59c-43b1-9c0a-011635edd419-model-cache\") pod \"distinct-2-w3-7a8fa2-kserve-677bb44d8d-fg55k\" (UID: \"e8701242-b59c-43b1-9c0a-011635edd419\") " pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-2-w3-7a8fa2-kserve-677bb44d8d-fg55k" Oct 30 18:47:26.286341 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:26.286314 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8701242-b59c-43b1-9c0a-011635edd419-tmp-dir\") pod \"distinct-2-w3-7a8fa2-kserve-677bb44d8d-fg55k\" (UID: \"e8701242-b59c-43b1-9c0a-011635edd419\") " pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-2-w3-7a8fa2-kserve-677bb44d8d-fg55k" Oct 30 18:47:26.286484 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:26.286466 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8701242-b59c-43b1-9c0a-011635edd419-tls-certs\") pod \"distinct-2-w3-7a8fa2-kserve-677bb44d8d-fg55k\" (UID: \"e8701242-b59c-43b1-9c0a-011635edd419\") " pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-2-w3-7a8fa2-kserve-677bb44d8d-fg55k" Oct 30 18:47:26.286681 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:26.286659 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r822d\" (UniqueName: \"kubernetes.io/projected/e8701242-b59c-43b1-9c0a-011635edd419-kube-api-access-r822d\") pod \"distinct-2-w3-7a8fa2-kserve-677bb44d8d-fg55k\" (UID: \"e8701242-b59c-43b1-9c0a-011635edd419\") " pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-2-w3-7a8fa2-kserve-677bb44d8d-fg55k" Oct 30 18:47:26.286780 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:26.286713 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8701242-b59c-43b1-9c0a-011635edd419-home\") pod \"distinct-2-w3-7a8fa2-kserve-677bb44d8d-fg55k\" (UID: \"e8701242-b59c-43b1-9c0a-011635edd419\") " pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-2-w3-7a8fa2-kserve-677bb44d8d-fg55k" Oct 30 18:47:26.391833 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:26.388791 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8701242-b59c-43b1-9c0a-011635edd419-tls-certs\") pod \"distinct-2-w3-7a8fa2-kserve-677bb44d8d-fg55k\" (UID: \"e8701242-b59c-43b1-9c0a-011635edd419\") " pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-2-w3-7a8fa2-kserve-677bb44d8d-fg55k" Oct 30 18:47:26.391833 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:26.388865 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r822d\" (UniqueName: \"kubernetes.io/projected/e8701242-b59c-43b1-9c0a-011635edd419-kube-api-access-r822d\") pod \"distinct-2-w3-7a8fa2-kserve-677bb44d8d-fg55k\" (UID: \"e8701242-b59c-43b1-9c0a-011635edd419\") " pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-2-w3-7a8fa2-kserve-677bb44d8d-fg55k" Oct 30 18:47:26.391833 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:26.388913 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8701242-b59c-43b1-9c0a-011635edd419-home\") pod \"distinct-2-w3-7a8fa2-kserve-677bb44d8d-fg55k\" (UID: \"e8701242-b59c-43b1-9c0a-011635edd419\") " pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-2-w3-7a8fa2-kserve-677bb44d8d-fg55k" Oct 30 18:47:26.391833 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:26.389046 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8701242-b59c-43b1-9c0a-011635edd419-dshm\") pod \"distinct-2-w3-7a8fa2-kserve-677bb44d8d-fg55k\" (UID: \"e8701242-b59c-43b1-9c0a-011635edd419\") " pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-2-w3-7a8fa2-kserve-677bb44d8d-fg55k" Oct 30 18:47:26.391833 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:26.389100 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8701242-b59c-43b1-9c0a-011635edd419-model-cache\") pod \"distinct-2-w3-7a8fa2-kserve-677bb44d8d-fg55k\" (UID: \"e8701242-b59c-43b1-9c0a-011635edd419\") " pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-2-w3-7a8fa2-kserve-677bb44d8d-fg55k" Oct 30 18:47:26.391833 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:26.389125 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8701242-b59c-43b1-9c0a-011635edd419-tmp-dir\") pod \"distinct-2-w3-7a8fa2-kserve-677bb44d8d-fg55k\" (UID: \"e8701242-b59c-43b1-9c0a-011635edd419\") " pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-2-w3-7a8fa2-kserve-677bb44d8d-fg55k" Oct 30 18:47:26.391833 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:26.389630 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8701242-b59c-43b1-9c0a-011635edd419-tmp-dir\") pod \"distinct-2-w3-7a8fa2-kserve-677bb44d8d-fg55k\" (UID: \"e8701242-b59c-43b1-9c0a-011635edd419\") " pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-2-w3-7a8fa2-kserve-677bb44d8d-fg55k" Oct 30 18:47:26.399529 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:26.399498 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8701242-b59c-43b1-9c0a-011635edd419-home\") pod \"distinct-2-w3-7a8fa2-kserve-677bb44d8d-fg55k\" (UID: \"e8701242-b59c-43b1-9c0a-011635edd419\") " pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-2-w3-7a8fa2-kserve-677bb44d8d-fg55k" Oct 30 18:47:26.402704 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:26.402678 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8701242-b59c-43b1-9c0a-011635edd419-dshm\") pod \"distinct-2-w3-7a8fa2-kserve-677bb44d8d-fg55k\" (UID: \"e8701242-b59c-43b1-9c0a-011635edd419\") " pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-2-w3-7a8fa2-kserve-677bb44d8d-fg55k" Oct 30 18:47:26.404606 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:26.404133 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8701242-b59c-43b1-9c0a-011635edd419-model-cache\") pod \"distinct-2-w3-7a8fa2-kserve-677bb44d8d-fg55k\" (UID: \"e8701242-b59c-43b1-9c0a-011635edd419\") " pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-2-w3-7a8fa2-kserve-677bb44d8d-fg55k" Oct 30 18:47:26.425201 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:26.420797 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r822d\" (UniqueName: \"kubernetes.io/projected/e8701242-b59c-43b1-9c0a-011635edd419-kube-api-access-r822d\") pod \"distinct-2-w3-7a8fa2-kserve-677bb44d8d-fg55k\" (UID: \"e8701242-b59c-43b1-9c0a-011635edd419\") " pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-2-w3-7a8fa2-kserve-677bb44d8d-fg55k" Oct 30 18:47:26.442204 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:26.437726 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8701242-b59c-43b1-9c0a-011635edd419-tls-certs\") pod \"distinct-2-w3-7a8fa2-kserve-677bb44d8d-fg55k\" (UID: \"e8701242-b59c-43b1-9c0a-011635edd419\") " pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-2-w3-7a8fa2-kserve-677bb44d8d-fg55k" Oct 30 18:47:26.508196 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:26.508070 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-2-w3-7a8fa2-kserve-677bb44d8d-fg55k" Oct 30 18:47:26.799302 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:26.799253 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-7a8fa2/distinct-2-w3-7a8fa2-kserve-677bb44d8d-fg55k"] Oct 30 18:47:26.800868 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:47:26.800837 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode8701242_b59c_43b1_9c0a_011635edd419.slice/crio-0a8181a958839877f9bea10b17db1ab5ff7f14819ee0ec5d7b01f16ccf8594fe WatchSource:0}: Error finding container 0a8181a958839877f9bea10b17db1ab5ff7f14819ee0ec5d7b01f16ccf8594fe: Status 404 returned error can't find the container with id 0a8181a958839877f9bea10b17db1ab5ff7f14819ee0ec5d7b01f16ccf8594fe Oct 30 18:47:27.026589 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:27.026553 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:27.663868 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:27.663830 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-2-w3-7a8fa2-kserve-677bb44d8d-fg55k" event={"ID":"e8701242-b59c-43b1-9c0a-011635edd419","Type":"ContainerStarted","Data":"da8c7e67cc7d7dbc66bb234713ee5a5e9a629cf12bd94737827fc249f1996cce"} Oct 30 18:47:27.663868 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:27.663879 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-2-w3-7a8fa2-kserve-677bb44d8d-fg55k" event={"ID":"e8701242-b59c-43b1-9c0a-011635edd419","Type":"ContainerStarted","Data":"0a8181a958839877f9bea10b17db1ab5ff7f14819ee0ec5d7b01f16ccf8594fe"} Oct 30 18:47:27.686078 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:27.686033 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-2-w3-7a8fa2-kserve-677bb44d8d-fg55k" podStartSLOduration=1.686018656 podStartE2EDuration="1.686018656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:47:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:47:27.684456568 +0000 UTC m=+815.213487706" watchObservedRunningTime="2026-10-30 18:47:27.686018656 +0000 UTC m=+815.215049833" Oct 30 18:47:28.015490 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:28.015394 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:28.509122 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:28.509092 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-2-w3-7a8fa2-kserve-677bb44d8d-fg55k" Oct 30 18:47:28.919692 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:28.919657 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:30.100847 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:30.100805 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:31.162936 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:31.162902 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:32.193074 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:32.193036 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:32.260210 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:32.260179 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-drlj6"] Oct 30 18:47:32.264428 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:32.264410 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-drlj6" Oct 30 18:47:32.307097 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:32.307070 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-drlj6"] Oct 30 18:47:32.371336 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:32.371302 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ae98fd5-ff16-44e3-80aa-139438c86c78-istio-data\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-drlj6\" (UID: \"1ae98fd5-ff16-44e3-80aa-139438c86c78\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-drlj6" Oct 30 18:47:32.371500 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:32.371350 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ae98fd5-ff16-44e3-80aa-139438c86c78-istiod-ca-cert\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-drlj6\" (UID: \"1ae98fd5-ff16-44e3-80aa-139438c86c78\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-drlj6" Oct 30 18:47:32.371500 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:32.371429 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ae98fd5-ff16-44e3-80aa-139438c86c78-credential-socket\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-drlj6\" (UID: \"1ae98fd5-ff16-44e3-80aa-139438c86c78\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-drlj6" Oct 30 18:47:32.371500 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:32.371448 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87bxp\" (UniqueName: \"kubernetes.io/projected/1ae98fd5-ff16-44e3-80aa-139438c86c78-kube-api-access-87bxp\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-drlj6\" (UID: \"1ae98fd5-ff16-44e3-80aa-139438c86c78\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-drlj6" Oct 30 18:47:32.371641 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:32.371611 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ae98fd5-ff16-44e3-80aa-139438c86c78-workload-certs\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-drlj6\" (UID: \"1ae98fd5-ff16-44e3-80aa-139438c86c78\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-drlj6" Oct 30 18:47:32.371689 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:32.371672 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ae98fd5-ff16-44e3-80aa-139438c86c78-workload-socket\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-drlj6\" (UID: \"1ae98fd5-ff16-44e3-80aa-139438c86c78\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-drlj6" Oct 30 18:47:32.371773 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:32.371754 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ae98fd5-ff16-44e3-80aa-139438c86c78-istio-token\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-drlj6\" (UID: \"1ae98fd5-ff16-44e3-80aa-139438c86c78\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-drlj6" Oct 30 18:47:32.371918 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:32.371902 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ae98fd5-ff16-44e3-80aa-139438c86c78-istio-podinfo\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-drlj6\" (UID: \"1ae98fd5-ff16-44e3-80aa-139438c86c78\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-drlj6" Oct 30 18:47:32.371982 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:32.371944 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ae98fd5-ff16-44e3-80aa-139438c86c78-istio-envoy\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-drlj6\" (UID: \"1ae98fd5-ff16-44e3-80aa-139438c86c78\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-drlj6" Oct 30 18:47:32.473187 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:32.473099 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ae98fd5-ff16-44e3-80aa-139438c86c78-istio-podinfo\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-drlj6\" (UID: \"1ae98fd5-ff16-44e3-80aa-139438c86c78\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-drlj6" Oct 30 18:47:32.473187 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:32.473137 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ae98fd5-ff16-44e3-80aa-139438c86c78-istio-envoy\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-drlj6\" (UID: \"1ae98fd5-ff16-44e3-80aa-139438c86c78\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-drlj6" Oct 30 18:47:32.473371 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:32.473252 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ae98fd5-ff16-44e3-80aa-139438c86c78-istio-data\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-drlj6\" (UID: \"1ae98fd5-ff16-44e3-80aa-139438c86c78\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-drlj6" Oct 30 18:47:32.473371 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:32.473296 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ae98fd5-ff16-44e3-80aa-139438c86c78-istiod-ca-cert\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-drlj6\" (UID: \"1ae98fd5-ff16-44e3-80aa-139438c86c78\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-drlj6" Oct 30 18:47:32.473371 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:32.473332 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ae98fd5-ff16-44e3-80aa-139438c86c78-credential-socket\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-drlj6\" (UID: \"1ae98fd5-ff16-44e3-80aa-139438c86c78\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-drlj6" Oct 30 18:47:32.473371 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:32.473351 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87bxp\" (UniqueName: \"kubernetes.io/projected/1ae98fd5-ff16-44e3-80aa-139438c86c78-kube-api-access-87bxp\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-drlj6\" (UID: \"1ae98fd5-ff16-44e3-80aa-139438c86c78\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-drlj6" Oct 30 18:47:32.473567 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:32.473399 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ae98fd5-ff16-44e3-80aa-139438c86c78-workload-certs\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-drlj6\" (UID: \"1ae98fd5-ff16-44e3-80aa-139438c86c78\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-drlj6" Oct 30 18:47:32.473567 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:32.473444 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ae98fd5-ff16-44e3-80aa-139438c86c78-workload-socket\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-drlj6\" (UID: \"1ae98fd5-ff16-44e3-80aa-139438c86c78\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-drlj6" Oct 30 18:47:32.473567 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:32.473469 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ae98fd5-ff16-44e3-80aa-139438c86c78-istio-token\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-drlj6\" (UID: \"1ae98fd5-ff16-44e3-80aa-139438c86c78\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-drlj6" Oct 30 18:47:32.473713 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:32.473701 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ae98fd5-ff16-44e3-80aa-139438c86c78-istio-data\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-drlj6\" (UID: \"1ae98fd5-ff16-44e3-80aa-139438c86c78\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-drlj6" Oct 30 18:47:32.473783 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:32.473756 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ae98fd5-ff16-44e3-80aa-139438c86c78-credential-socket\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-drlj6\" (UID: \"1ae98fd5-ff16-44e3-80aa-139438c86c78\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-drlj6" Oct 30 18:47:32.473961 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:32.473944 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ae98fd5-ff16-44e3-80aa-139438c86c78-workload-socket\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-drlj6\" (UID: \"1ae98fd5-ff16-44e3-80aa-139438c86c78\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-drlj6" Oct 30 18:47:32.474059 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:32.474004 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ae98fd5-ff16-44e3-80aa-139438c86c78-workload-certs\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-drlj6\" (UID: \"1ae98fd5-ff16-44e3-80aa-139438c86c78\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-drlj6" Oct 30 18:47:32.474411 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:32.474389 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ae98fd5-ff16-44e3-80aa-139438c86c78-istiod-ca-cert\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-drlj6\" (UID: \"1ae98fd5-ff16-44e3-80aa-139438c86c78\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-drlj6" Oct 30 18:47:32.475844 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:32.475825 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ae98fd5-ff16-44e3-80aa-139438c86c78-istio-podinfo\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-drlj6\" (UID: \"1ae98fd5-ff16-44e3-80aa-139438c86c78\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-drlj6" Oct 30 18:47:32.476075 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:32.476058 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ae98fd5-ff16-44e3-80aa-139438c86c78-istio-envoy\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-drlj6\" (UID: \"1ae98fd5-ff16-44e3-80aa-139438c86c78\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-drlj6" Oct 30 18:47:32.485876 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:32.485850 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ae98fd5-ff16-44e3-80aa-139438c86c78-istio-token\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-drlj6\" (UID: \"1ae98fd5-ff16-44e3-80aa-139438c86c78\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-drlj6" Oct 30 18:47:32.486095 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:32.486071 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87bxp\" (UniqueName: \"kubernetes.io/projected/1ae98fd5-ff16-44e3-80aa-139438c86c78-kube-api-access-87bxp\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-drlj6\" (UID: \"1ae98fd5-ff16-44e3-80aa-139438c86c78\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-drlj6" Oct 30 18:47:32.578605 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:32.578564 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-drlj6" Oct 30 18:47:32.870031 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:32.870007 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-drlj6"] Oct 30 18:47:32.877934 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:47:32.877889 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ae98fd5_ff16_44e3_80aa_139438c86c78.slice/crio-747b2286bc9d222fa9f90370687483162138ca9657021df88a184f80a9f5e881 WatchSource:0}: Error finding container 747b2286bc9d222fa9f90370687483162138ca9657021df88a184f80a9f5e881: Status 404 returned error can't find the container with id 747b2286bc9d222fa9f90370687483162138ca9657021df88a184f80a9f5e881 Oct 30 18:47:32.880578 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:32.880540 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:47:32.880659 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:32.880627 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:47:32.880697 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:32.880670 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:47:33.229477 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:33.229393 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:33.718544 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:33.718508 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-drlj6" event={"ID":"1ae98fd5-ff16-44e3-80aa-139438c86c78","Type":"ContainerStarted","Data":"31b0e3829260bd99c17c01905cb36463411dd72de7203f1e0db0ded1f8915fdc"} Oct 30 18:47:33.718724 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:33.718562 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-drlj6" event={"ID":"1ae98fd5-ff16-44e3-80aa-139438c86c78","Type":"ContainerStarted","Data":"747b2286bc9d222fa9f90370687483162138ca9657021df88a184f80a9f5e881"} Oct 30 18:47:33.744936 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:33.744886 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-drlj6" podStartSLOduration=1.74487305 podStartE2EDuration="1.74487305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:47:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:47:33.741982967 +0000 UTC m=+821.271014107" watchObservedRunningTime="2026-10-30 18:47:33.74487305 +0000 UTC m=+821.273904249" Oct 30 18:47:34.188821 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:34.188782 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:34.580168 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:34.579679 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-drlj6" Oct 30 18:47:34.582413 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:34.582381 6960 patch_prober.go:29] interesting pod/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-drlj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 18:47:34.582552 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:34.582436 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-drlj6" podUID="1ae98fd5-ff16-44e3-80aa-139438c86c78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 18:47:35.137356 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:35.137312 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:35.580179 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:35.579474 6960 patch_prober.go:29] interesting pod/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-drlj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 18:47:35.580179 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:35.579535 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-drlj6" podUID="1ae98fd5-ff16-44e3-80aa-139438c86c78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 18:47:36.100027 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:36.099988 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:36.508983 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:36.508909 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-2-w3-7a8fa2-kserve-677bb44d8d-fg55k" Oct 30 18:47:36.522613 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:36.522569 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-2-w3-7a8fa2-kserve-677bb44d8d-fg55k" Oct 30 18:47:36.587133 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:36.587101 6960 patch_prober.go:29] interesting pod/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-drlj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 18:47:36.587657 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:36.587174 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-drlj6" podUID="1ae98fd5-ff16-44e3-80aa-139438c86c78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 18:47:36.614697 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:36.614669 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:36.807855 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:36.807828 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-2-w3-7a8fa2-kserve-677bb44d8d-fg55k" Oct 30 18:47:37.134231 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:37.134130 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:37.579516 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:37.579487 6960 patch_prober.go:29] interesting pod/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-drlj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 18:47:37.580691 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:37.580659 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-drlj6" podUID="1ae98fd5-ff16-44e3-80aa-139438c86c78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 18:47:38.207808 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:38.207769 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:38.579928 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:38.579895 6960 patch_prober.go:29] interesting pod/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-drlj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 18:47:38.580257 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:38.580227 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-drlj6" podUID="1ae98fd5-ff16-44e3-80aa-139438c86c78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 18:47:39.179769 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:39.179733 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:39.579341 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:39.579301 6960 patch_prober.go:29] interesting pod/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-drlj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 18:47:39.579818 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:39.579371 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-drlj6" podUID="1ae98fd5-ff16-44e3-80aa-139438c86c78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 18:47:40.141181 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:40.141119 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:40.583109 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:40.583082 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-drlj6" Oct 30 18:47:40.583504 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:40.583456 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-drlj6" Oct 30 18:47:40.584130 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:40.584114 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-drlj6" Oct 30 18:47:41.170991 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:41.170951 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:42.239171 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:42.239036 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-h72dd"] Oct 30 18:47:42.456488 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:42.456367 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-h72dd"] Oct 30 18:47:42.513227 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:42.459959 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-h72dd" Oct 30 18:47:42.597181 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:42.596747 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b-istiod-ca-cert\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-h72dd\" (UID: \"4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-h72dd" Oct 30 18:47:42.597181 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:42.596803 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxbnk\" (UniqueName: \"kubernetes.io/projected/4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b-kube-api-access-vxbnk\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-h72dd\" (UID: \"4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-h72dd" Oct 30 18:47:42.597181 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:42.596835 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b-istio-token\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-h72dd\" (UID: \"4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-h72dd" Oct 30 18:47:42.597181 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:42.596862 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b-istio-envoy\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-h72dd\" (UID: \"4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-h72dd" Oct 30 18:47:42.597181 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:42.596891 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b-workload-socket\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-h72dd\" (UID: \"4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-h72dd" Oct 30 18:47:42.597181 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:42.596942 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b-istio-podinfo\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-h72dd\" (UID: \"4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-h72dd" Oct 30 18:47:42.597181 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:42.597054 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b-istio-data\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-h72dd\" (UID: \"4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-h72dd" Oct 30 18:47:42.597181 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:42.597119 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b-credential-socket\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-h72dd\" (UID: \"4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-h72dd" Oct 30 18:47:42.597181 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:42.597190 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b-workload-certs\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-h72dd\" (UID: \"4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-h72dd" Oct 30 18:47:42.700285 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:42.699249 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b-workload-certs\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-h72dd\" (UID: \"4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-h72dd" Oct 30 18:47:42.700285 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:42.699398 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b-istiod-ca-cert\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-h72dd\" (UID: \"4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-h72dd" Oct 30 18:47:42.700285 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:42.699441 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxbnk\" (UniqueName: \"kubernetes.io/projected/4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b-kube-api-access-vxbnk\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-h72dd\" (UID: \"4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-h72dd" Oct 30 18:47:42.700285 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:42.699490 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b-istio-token\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-h72dd\" (UID: \"4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-h72dd" Oct 30 18:47:42.700285 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:42.699517 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b-istio-envoy\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-h72dd\" (UID: \"4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-h72dd" Oct 30 18:47:42.700285 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:42.699568 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b-workload-socket\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-h72dd\" (UID: \"4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-h72dd" Oct 30 18:47:42.714248 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:42.704223 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b-istio-podinfo\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-h72dd\" (UID: \"4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-h72dd" Oct 30 18:47:42.714248 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:42.704447 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b-istio-data\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-h72dd\" (UID: \"4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-h72dd" Oct 30 18:47:42.714248 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:42.704553 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b-credential-socket\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-h72dd\" (UID: \"4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-h72dd" Oct 30 18:47:42.714248 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:42.705291 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b-credential-socket\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-h72dd\" (UID: \"4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-h72dd" Oct 30 18:47:42.714248 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:42.705993 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b-workload-certs\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-h72dd\" (UID: \"4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-h72dd" Oct 30 18:47:42.714248 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:42.710360 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b-workload-socket\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-h72dd\" (UID: \"4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-h72dd" Oct 30 18:47:42.714248 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:42.710841 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b-istio-data\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-h72dd\" (UID: \"4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-h72dd" Oct 30 18:47:42.714248 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:42.713743 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b-istiod-ca-cert\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-h72dd\" (UID: \"4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-h72dd" Oct 30 18:47:42.756972 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:42.756939 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b-istio-envoy\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-h72dd\" (UID: \"4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-h72dd" Oct 30 18:47:42.764817 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:42.764710 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b-istio-podinfo\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-h72dd\" (UID: \"4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-h72dd" Oct 30 18:47:42.765558 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:42.765531 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b-istio-token\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-h72dd\" (UID: \"4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-h72dd" Oct 30 18:47:42.786259 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:42.786124 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:42.788013 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:42.787993 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxbnk\" (UniqueName: \"kubernetes.io/projected/4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b-kube-api-access-vxbnk\") pod \"e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-h72dd\" (UID: \"4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b\") " pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-h72dd" Oct 30 18:47:42.815184 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:42.815034 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-h72dd" Oct 30 18:47:43.545782 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:43.544979 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-2-w2-8a92cc-kserve-6cc4b9c974-n7cd2" podUID="e5fd8f33-8662-4f90-9817-e601f694bf1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:47:43.629223 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:43.622605 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-aa6736/premium-w1-aa6736-kserve-6f986b66f7-nkfrg" podUID="84ca40c5-7033-48c4-bcec-e9ac7c034a4e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:47:43.705169 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:43.704136 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-h72dd"] Oct 30 18:47:43.754785 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:43.753396 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-2-w3-7a8fa2-kserve-677bb44d8d-fg55k" podUID="e8701242-b59c-43b1-9c0a-011635edd419" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:47:43.786754 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:43.785113 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:47:43.786754 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:43.785228 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:47:43.786754 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:43.785275 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:47:43.818367 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:47:43.818333 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cb85a2d_46a2_43e3_8d13_4cfc15f48c5b.slice/crio-5a3c8824468116a1a123f556e34b4a5ac822c45e113ef345d8445183db8a2fb3 WatchSource:0}: Error finding container 5a3c8824468116a1a123f556e34b4a5ac822c45e113ef345d8445183db8a2fb3: Status 404 returned error can't find the container with id 5a3c8824468116a1a123f556e34b4a5ac822c45e113ef345d8445183db8a2fb3 Oct 30 18:47:43.950235 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:43.948089 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-h72dd" event={"ID":"4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b","Type":"ContainerStarted","Data":"5a3c8824468116a1a123f556e34b4a5ac822c45e113ef345d8445183db8a2fb3"} Oct 30 18:47:45.015881 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:45.015442 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-h72dd" event={"ID":"4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b","Type":"ContainerStarted","Data":"ddb3577403de659337d418cae169967e2fbdabb6918860b2f17f55185ba3ee90"} Oct 30 18:47:45.069210 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:45.068221 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-h72dd" podStartSLOduration=3.068202961 podStartE2EDuration="3.068202961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:47:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:47:45.063827895 +0000 UTC m=+832.592859037" watchObservedRunningTime="2026-10-30 18:47:45.068202961 +0000 UTC m=+832.597234099" Oct 30 18:47:45.696774 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:45.696745 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:45.820291 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:45.820263 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-h72dd" Oct 30 18:47:45.821689 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:45.821665 6960 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-h72dd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 18:47:45.821854 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:45.821718 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-h72dd" podUID="4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 18:47:46.725504 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:46.725470 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:46.820545 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:46.820511 6960 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-h72dd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 18:47:46.820691 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:46.820573 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-h72dd" podUID="4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 18:47:47.820333 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:47.820299 6960 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-h72dd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 18:47:47.820722 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:47.820369 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-h72dd" podUID="4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 18:47:47.937626 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:47.937594 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:48.820509 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:48.820474 6960 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-h72dd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 18:47:48.820898 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:48.820546 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-h72dd" podUID="4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 18:47:48.886449 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:48.886390 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:49.820162 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:49.820114 6960 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-h72dd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 18:47:49.820339 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:49.820195 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-h72dd" podUID="4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 18:47:49.827301 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:49.827272 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:50.383597 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:50.383558 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:50.820208 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:50.820175 6960 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-h72dd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 18:47:50.820458 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:50.820428 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-h72dd" podUID="4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 18:47:50.911526 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:50.911495 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:51.820002 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:51.819971 6960 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-h72dd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 18:47:51.820219 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:51.820023 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-h72dd" podUID="4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 18:47:52.150822 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:52.150739 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:52.824368 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:52.824342 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-h72dd" Oct 30 18:47:52.824701 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:52.824683 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-h72dd" Oct 30 18:47:52.825424 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:52.825409 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-h72dd" Oct 30 18:47:53.418651 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:53.418600 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:53.932095 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:47:53.932065 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 18:47:54.598943 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:54.598913 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:55.519786 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:55.519746 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:56.446858 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:56.446816 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:57.313410 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:57.313378 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:58.219997 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:58.219963 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:59.135763 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:47:59.135727 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:00.196764 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:00.196720 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:02.462096 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:02.461489 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7a8fa2/premium-w3-7a8fa2-kserve-6b6c5f6fb8-n5wtq" podUID="f51fd746-cd85-4608-96a6-f54afa0ec34d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:48:02.542342 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:02.542293 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/simulator-w2-8a92cc-kserve-fb77d84bc-nqf9q" podUID="ce70492e-c727-4dc1-b2ee-bf278d29a45b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:48:02.624453 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:02.623845 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-aa6736/simulator-w1-aa6736-kserve-76985dcd5b-njvnn" podUID="943f7840-8d54-4845-b7ec-c0da72ba9c67" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:48:02.776250 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:02.767714 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-2-w3-7a8fa2-kserve-677bb44d8d-fg55k" podUID="e8701242-b59c-43b1-9c0a-011635edd419" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 30 18:48:03.566580 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:03.566538 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:05.063195 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:05.063139 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:05.857614 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:05.857578 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:07.086390 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:07.086340 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:08.314112 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:08.314083 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:09.531826 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:09.531789 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:09.978223 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:09.978194 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-c2m5d"] Oct 30 18:48:09.982203 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:09.982187 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-c2m5d" Oct 30 18:48:10.000420 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:10.000373 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-c2m5d"] Oct 30 18:48:10.137082 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:10.137052 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/090b2a4e-323a-4a3c-9a5b-165df26c5b56-istio-data\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-c2m5d\" (UID: \"090b2a4e-323a-4a3c-9a5b-165df26c5b56\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-c2m5d" Oct 30 18:48:10.137278 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:10.137112 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/090b2a4e-323a-4a3c-9a5b-165df26c5b56-istio-podinfo\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-c2m5d\" (UID: \"090b2a4e-323a-4a3c-9a5b-165df26c5b56\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-c2m5d" Oct 30 18:48:10.137278 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:10.137261 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/090b2a4e-323a-4a3c-9a5b-165df26c5b56-workload-certs\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-c2m5d\" (UID: \"090b2a4e-323a-4a3c-9a5b-165df26c5b56\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-c2m5d" Oct 30 18:48:10.137400 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:10.137342 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsj5n\" (UniqueName: \"kubernetes.io/projected/090b2a4e-323a-4a3c-9a5b-165df26c5b56-kube-api-access-tsj5n\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-c2m5d\" (UID: \"090b2a4e-323a-4a3c-9a5b-165df26c5b56\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-c2m5d" Oct 30 18:48:10.137435 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:10.137422 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/090b2a4e-323a-4a3c-9a5b-165df26c5b56-istio-envoy\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-c2m5d\" (UID: \"090b2a4e-323a-4a3c-9a5b-165df26c5b56\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-c2m5d" Oct 30 18:48:10.137505 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:10.137492 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/090b2a4e-323a-4a3c-9a5b-165df26c5b56-istio-token\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-c2m5d\" (UID: \"090b2a4e-323a-4a3c-9a5b-165df26c5b56\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-c2m5d" Oct 30 18:48:10.137545 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:10.137536 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/090b2a4e-323a-4a3c-9a5b-165df26c5b56-workload-socket\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-c2m5d\" (UID: \"090b2a4e-323a-4a3c-9a5b-165df26c5b56\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-c2m5d" Oct 30 18:48:10.137643 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:10.137631 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/090b2a4e-323a-4a3c-9a5b-165df26c5b56-istiod-ca-cert\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-c2m5d\" (UID: \"090b2a4e-323a-4a3c-9a5b-165df26c5b56\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-c2m5d" Oct 30 18:48:10.137678 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:10.137653 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/090b2a4e-323a-4a3c-9a5b-165df26c5b56-credential-socket\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-c2m5d\" (UID: \"090b2a4e-323a-4a3c-9a5b-165df26c5b56\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-c2m5d" Oct 30 18:48:10.238666 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:10.238586 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/090b2a4e-323a-4a3c-9a5b-165df26c5b56-istiod-ca-cert\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-c2m5d\" (UID: \"090b2a4e-323a-4a3c-9a5b-165df26c5b56\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-c2m5d" Oct 30 18:48:10.238666 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:10.238623 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/090b2a4e-323a-4a3c-9a5b-165df26c5b56-credential-socket\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-c2m5d\" (UID: \"090b2a4e-323a-4a3c-9a5b-165df26c5b56\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-c2m5d" Oct 30 18:48:10.238896 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:10.238672 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/090b2a4e-323a-4a3c-9a5b-165df26c5b56-istio-data\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-c2m5d\" (UID: \"090b2a4e-323a-4a3c-9a5b-165df26c5b56\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-c2m5d" Oct 30 18:48:10.238896 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:10.238717 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/090b2a4e-323a-4a3c-9a5b-165df26c5b56-istio-podinfo\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-c2m5d\" (UID: \"090b2a4e-323a-4a3c-9a5b-165df26c5b56\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-c2m5d" Oct 30 18:48:10.238896 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:10.238751 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/090b2a4e-323a-4a3c-9a5b-165df26c5b56-workload-certs\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-c2m5d\" (UID: \"090b2a4e-323a-4a3c-9a5b-165df26c5b56\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-c2m5d" Oct 30 18:48:10.238896 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:10.238798 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsj5n\" (UniqueName: \"kubernetes.io/projected/090b2a4e-323a-4a3c-9a5b-165df26c5b56-kube-api-access-tsj5n\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-c2m5d\" (UID: \"090b2a4e-323a-4a3c-9a5b-165df26c5b56\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-c2m5d" Oct 30 18:48:10.238896 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:10.238884 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/090b2a4e-323a-4a3c-9a5b-165df26c5b56-istio-envoy\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-c2m5d\" (UID: \"090b2a4e-323a-4a3c-9a5b-165df26c5b56\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-c2m5d" Oct 30 18:48:10.239136 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:10.238910 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/090b2a4e-323a-4a3c-9a5b-165df26c5b56-istio-token\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-c2m5d\" (UID: \"090b2a4e-323a-4a3c-9a5b-165df26c5b56\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-c2m5d" Oct 30 18:48:10.239136 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:10.238961 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/090b2a4e-323a-4a3c-9a5b-165df26c5b56-workload-socket\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-c2m5d\" (UID: \"090b2a4e-323a-4a3c-9a5b-165df26c5b56\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-c2m5d" Oct 30 18:48:10.239272 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:10.239127 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/090b2a4e-323a-4a3c-9a5b-165df26c5b56-credential-socket\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-c2m5d\" (UID: \"090b2a4e-323a-4a3c-9a5b-165df26c5b56\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-c2m5d" Oct 30 18:48:10.239272 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:10.239218 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/090b2a4e-323a-4a3c-9a5b-165df26c5b56-istio-data\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-c2m5d\" (UID: \"090b2a4e-323a-4a3c-9a5b-165df26c5b56\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-c2m5d" Oct 30 18:48:10.239376 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:10.239267 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/090b2a4e-323a-4a3c-9a5b-165df26c5b56-workload-certs\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-c2m5d\" (UID: \"090b2a4e-323a-4a3c-9a5b-165df26c5b56\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-c2m5d" Oct 30 18:48:10.239376 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:10.239334 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/090b2a4e-323a-4a3c-9a5b-165df26c5b56-workload-socket\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-c2m5d\" (UID: \"090b2a4e-323a-4a3c-9a5b-165df26c5b56\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-c2m5d" Oct 30 18:48:10.239613 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:10.239591 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/090b2a4e-323a-4a3c-9a5b-165df26c5b56-istiod-ca-cert\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-c2m5d\" (UID: \"090b2a4e-323a-4a3c-9a5b-165df26c5b56\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-c2m5d" Oct 30 18:48:10.241430 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:10.241409 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/090b2a4e-323a-4a3c-9a5b-165df26c5b56-istio-podinfo\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-c2m5d\" (UID: \"090b2a4e-323a-4a3c-9a5b-165df26c5b56\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-c2m5d" Oct 30 18:48:10.241556 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:10.241452 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/090b2a4e-323a-4a3c-9a5b-165df26c5b56-istio-envoy\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-c2m5d\" (UID: \"090b2a4e-323a-4a3c-9a5b-165df26c5b56\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-c2m5d" Oct 30 18:48:10.251347 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:10.251322 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsj5n\" (UniqueName: \"kubernetes.io/projected/090b2a4e-323a-4a3c-9a5b-165df26c5b56-kube-api-access-tsj5n\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-c2m5d\" (UID: \"090b2a4e-323a-4a3c-9a5b-165df26c5b56\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-c2m5d" Oct 30 18:48:10.252315 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:10.252291 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/090b2a4e-323a-4a3c-9a5b-165df26c5b56-istio-token\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-c2m5d\" (UID: \"090b2a4e-323a-4a3c-9a5b-165df26c5b56\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-c2m5d" Oct 30 18:48:10.298684 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:10.298658 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-c2m5d" Oct 30 18:48:10.479343 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:10.479296 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:10.602840 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:10.602218 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-c2m5d"] Oct 30 18:48:10.605582 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:48:10.605553 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod090b2a4e_323a_4a3c_9a5b_165df26c5b56.slice/crio-c720afc5ba5ef50998fbc92999eab4b531664b064f98f3aa9e4dec67e9d235ad WatchSource:0}: Error finding container c720afc5ba5ef50998fbc92999eab4b531664b064f98f3aa9e4dec67e9d235ad: Status 404 returned error can't find the container with id c720afc5ba5ef50998fbc92999eab4b531664b064f98f3aa9e4dec67e9d235ad Oct 30 18:48:10.608488 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:10.608452 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:48:10.608573 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:10.608535 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:48:10.608610 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:10.608574 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:48:11.492231 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:11.492197 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-c2m5d" event={"ID":"090b2a4e-323a-4a3c-9a5b-165df26c5b56","Type":"ContainerStarted","Data":"a50a647368a401a5e583a95ab90cb812dfe50d2f09364f5ef255fdefa4345aa8"} Oct 30 18:48:11.492415 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:11.492248 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-c2m5d" event={"ID":"090b2a4e-323a-4a3c-9a5b-165df26c5b56","Type":"ContainerStarted","Data":"c720afc5ba5ef50998fbc92999eab4b531664b064f98f3aa9e4dec67e9d235ad"} Oct 30 18:48:11.518426 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:11.518377 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-c2m5d" podStartSLOduration=2.518362593 podStartE2EDuration="2.518362593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:48:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:48:11.515816599 +0000 UTC m=+859.044847734" watchObservedRunningTime="2026-10-30 18:48:11.518362593 +0000 UTC m=+859.047393731" Oct 30 18:48:11.520865 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:11.520838 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:12.299017 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:12.298985 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-c2m5d" Oct 30 18:48:12.300458 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:12.300430 6960 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-c2m5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 18:48:12.300571 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:12.300505 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-c2m5d" podUID="090b2a4e-323a-4a3c-9a5b-165df26c5b56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 18:48:12.592836 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:12.592798 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:13.300062 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:13.300027 6960 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-c2m5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 18:48:13.300505 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:13.300088 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-c2m5d" podUID="090b2a4e-323a-4a3c-9a5b-165df26c5b56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 18:48:13.549190 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:13.549123 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:14.306567 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:14.306530 6960 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-c2m5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 18:48:14.307276 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:14.307247 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-c2m5d" podUID="090b2a4e-323a-4a3c-9a5b-165df26c5b56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 18:48:14.561674 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:14.561583 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:14.728170 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:14.721976 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:15.255970 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:15.255878 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:15.417532 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:15.417491 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:15.441940 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:15.441909 6960 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-c2m5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 18:48:15.442227 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:15.442196 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-c2m5d" podUID="090b2a4e-323a-4a3c-9a5b-165df26c5b56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 18:48:16.304658 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:16.304605 6960 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-c2m5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 18:48:16.304984 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:16.304938 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-c2m5d" podUID="090b2a4e-323a-4a3c-9a5b-165df26c5b56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 18:48:16.603244 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:16.601121 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-aa6736/distinct-w1-aa6736-kserve-b86d7667b-smcbz" podUID="58c23db7-11cb-4ce2-a67e-fe698e89c71d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 18:48:16.778567 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:16.778526 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:17.315700 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:17.315667 6960 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-c2m5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 18:48:17.315967 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:17.315938 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-c2m5d" podUID="090b2a4e-323a-4a3c-9a5b-165df26c5b56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 18:48:17.421606 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:17.421580 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-vjb5b"] Oct 30 18:48:17.427343 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:17.427318 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-vjb5b" Oct 30 18:48:17.461171 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:17.461118 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-vjb5b"] Oct 30 18:48:17.501949 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:17.501214 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/160390da-b35c-4030-9fff-d93afbcf9bad-credential-socket\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-vjb5b\" (UID: \"160390da-b35c-4030-9fff-d93afbcf9bad\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-vjb5b" Oct 30 18:48:17.501949 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:17.501369 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/160390da-b35c-4030-9fff-d93afbcf9bad-workload-socket\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-vjb5b\" (UID: \"160390da-b35c-4030-9fff-d93afbcf9bad\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-vjb5b" Oct 30 18:48:17.501949 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:17.501492 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/160390da-b35c-4030-9fff-d93afbcf9bad-istio-envoy\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-vjb5b\" (UID: \"160390da-b35c-4030-9fff-d93afbcf9bad\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-vjb5b" Oct 30 18:48:17.501949 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:17.501546 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/160390da-b35c-4030-9fff-d93afbcf9bad-istio-token\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-vjb5b\" (UID: \"160390da-b35c-4030-9fff-d93afbcf9bad\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-vjb5b" Oct 30 18:48:17.501949 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:17.501575 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/160390da-b35c-4030-9fff-d93afbcf9bad-istio-data\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-vjb5b\" (UID: \"160390da-b35c-4030-9fff-d93afbcf9bad\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-vjb5b" Oct 30 18:48:17.501949 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:17.501603 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/160390da-b35c-4030-9fff-d93afbcf9bad-workload-certs\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-vjb5b\" (UID: \"160390da-b35c-4030-9fff-d93afbcf9bad\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-vjb5b" Oct 30 18:48:17.501949 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:17.501671 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwvst\" (UniqueName: \"kubernetes.io/projected/160390da-b35c-4030-9fff-d93afbcf9bad-kube-api-access-lwvst\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-vjb5b\" (UID: \"160390da-b35c-4030-9fff-d93afbcf9bad\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-vjb5b" Oct 30 18:48:17.501949 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:17.501715 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/160390da-b35c-4030-9fff-d93afbcf9bad-istio-podinfo\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-vjb5b\" (UID: \"160390da-b35c-4030-9fff-d93afbcf9bad\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-vjb5b" Oct 30 18:48:17.501949 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:17.501750 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/160390da-b35c-4030-9fff-d93afbcf9bad-istiod-ca-cert\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-vjb5b\" (UID: \"160390da-b35c-4030-9fff-d93afbcf9bad\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-vjb5b" Oct 30 18:48:17.602898 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:17.602802 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwvst\" (UniqueName: \"kubernetes.io/projected/160390da-b35c-4030-9fff-d93afbcf9bad-kube-api-access-lwvst\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-vjb5b\" (UID: \"160390da-b35c-4030-9fff-d93afbcf9bad\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-vjb5b" Oct 30 18:48:17.603770 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:17.603740 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/160390da-b35c-4030-9fff-d93afbcf9bad-istio-podinfo\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-vjb5b\" (UID: \"160390da-b35c-4030-9fff-d93afbcf9bad\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-vjb5b" Oct 30 18:48:17.604351 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:17.604312 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/160390da-b35c-4030-9fff-d93afbcf9bad-istiod-ca-cert\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-vjb5b\" (UID: \"160390da-b35c-4030-9fff-d93afbcf9bad\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-vjb5b" Oct 30 18:48:17.604526 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:17.604511 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/160390da-b35c-4030-9fff-d93afbcf9bad-credential-socket\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-vjb5b\" (UID: \"160390da-b35c-4030-9fff-d93afbcf9bad\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-vjb5b" Oct 30 18:48:17.604733 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:17.604712 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/160390da-b35c-4030-9fff-d93afbcf9bad-workload-socket\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-vjb5b\" (UID: \"160390da-b35c-4030-9fff-d93afbcf9bad\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-vjb5b" Oct 30 18:48:17.604965 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:17.604924 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/160390da-b35c-4030-9fff-d93afbcf9bad-istio-envoy\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-vjb5b\" (UID: \"160390da-b35c-4030-9fff-d93afbcf9bad\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-vjb5b" Oct 30 18:48:17.605163 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:17.605128 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/160390da-b35c-4030-9fff-d93afbcf9bad-istio-token\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-vjb5b\" (UID: \"160390da-b35c-4030-9fff-d93afbcf9bad\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-vjb5b" Oct 30 18:48:17.605304 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:17.605289 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/160390da-b35c-4030-9fff-d93afbcf9bad-istio-data\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-vjb5b\" (UID: \"160390da-b35c-4030-9fff-d93afbcf9bad\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-vjb5b" Oct 30 18:48:17.605507 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:17.605491 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/160390da-b35c-4030-9fff-d93afbcf9bad-workload-certs\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-vjb5b\" (UID: \"160390da-b35c-4030-9fff-d93afbcf9bad\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-vjb5b" Oct 30 18:48:17.605689 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:17.605564 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/160390da-b35c-4030-9fff-d93afbcf9bad-credential-socket\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-vjb5b\" (UID: \"160390da-b35c-4030-9fff-d93afbcf9bad\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-vjb5b" Oct 30 18:48:17.606321 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:17.606279 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/160390da-b35c-4030-9fff-d93afbcf9bad-istio-data\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-vjb5b\" (UID: \"160390da-b35c-4030-9fff-d93afbcf9bad\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-vjb5b" Oct 30 18:48:17.607510 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:17.607428 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/160390da-b35c-4030-9fff-d93afbcf9bad-workload-certs\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-vjb5b\" (UID: \"160390da-b35c-4030-9fff-d93afbcf9bad\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-vjb5b" Oct 30 18:48:17.607746 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:17.607688 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/160390da-b35c-4030-9fff-d93afbcf9bad-workload-socket\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-vjb5b\" (UID: \"160390da-b35c-4030-9fff-d93afbcf9bad\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-vjb5b" Oct 30 18:48:17.608854 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:17.608338 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/160390da-b35c-4030-9fff-d93afbcf9bad-istiod-ca-cert\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-vjb5b\" (UID: \"160390da-b35c-4030-9fff-d93afbcf9bad\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-vjb5b" Oct 30 18:48:17.623467 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:17.621259 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/160390da-b35c-4030-9fff-d93afbcf9bad-istio-envoy\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-vjb5b\" (UID: \"160390da-b35c-4030-9fff-d93afbcf9bad\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-vjb5b" Oct 30 18:48:17.624623 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:17.624597 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/160390da-b35c-4030-9fff-d93afbcf9bad-istio-podinfo\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-vjb5b\" (UID: \"160390da-b35c-4030-9fff-d93afbcf9bad\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-vjb5b" Oct 30 18:48:17.631721 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:17.631693 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/160390da-b35c-4030-9fff-d93afbcf9bad-istio-token\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-vjb5b\" (UID: \"160390da-b35c-4030-9fff-d93afbcf9bad\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-vjb5b" Oct 30 18:48:17.651188 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:17.647521 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwvst\" (UniqueName: \"kubernetes.io/projected/160390da-b35c-4030-9fff-d93afbcf9bad-kube-api-access-lwvst\") pod \"e2e-worker-w2-8a92cc-openshift-default-849c89cccc-vjb5b\" (UID: \"160390da-b35c-4030-9fff-d93afbcf9bad\") " pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-vjb5b" Oct 30 18:48:17.757533 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:17.757506 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-vjb5b" Oct 30 18:48:18.080948 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:18.080915 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-vjb5b"] Oct 30 18:48:18.100931 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:48:18.100886 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod160390da_b35c_4030_9fff_d93afbcf9bad.slice/crio-64aecded2197f325b034fb18115e96ac6a5a0db79c41aee7f79e303a12576896 WatchSource:0}: Error finding container 64aecded2197f325b034fb18115e96ac6a5a0db79c41aee7f79e303a12576896: Status 404 returned error can't find the container with id 64aecded2197f325b034fb18115e96ac6a5a0db79c41aee7f79e303a12576896 Oct 30 18:48:18.104572 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:18.104531 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:48:18.104678 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:18.104633 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:48:18.104741 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:18.104678 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:48:18.300026 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:18.299975 6960 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-c2m5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 18:48:18.300209 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:18.300055 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-c2m5d" podUID="090b2a4e-323a-4a3c-9a5b-165df26c5b56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 18:48:18.459207 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:18.457672 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:18.606186 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:18.603796 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-vjb5b" event={"ID":"160390da-b35c-4030-9fff-d93afbcf9bad","Type":"ContainerStarted","Data":"af04992b96a297bcf36ebea4a331cfe69c8e368f7ccc7012f4e8851315113726"} Oct 30 18:48:18.606186 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:18.603863 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-vjb5b" event={"ID":"160390da-b35c-4030-9fff-d93afbcf9bad","Type":"ContainerStarted","Data":"64aecded2197f325b034fb18115e96ac6a5a0db79c41aee7f79e303a12576896"} Oct 30 18:48:18.618711 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:18.618668 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-vjb5b" podStartSLOduration=1.6186544889999999 podStartE2EDuration="1.618654489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:48:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:48:18.615683966 +0000 UTC m=+866.144715104" watchObservedRunningTime="2026-10-30 18:48:18.618654489 +0000 UTC m=+866.147685626" Oct 30 18:48:18.757729 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:18.757594 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-vjb5b" Oct 30 18:48:19.778202 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:19.777319 6960 patch_prober.go:29] interesting pod/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-vjb5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 18:48:19.778202 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:19.777393 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-vjb5b" podUID="160390da-b35c-4030-9fff-d93afbcf9bad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 18:48:20.020166 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:20.019576 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:20.342274 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:20.342246 6960 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-c2m5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:48:20.342511 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:20.342291 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-c2m5d" podUID="090b2a4e-323a-4a3c-9a5b-165df26c5b56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:48:20.432723 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:20.432689 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-c2m5d" Oct 30 18:48:20.433107 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:20.433084 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-c2m5d" Oct 30 18:48:20.433964 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:20.433948 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-c2m5d" Oct 30 18:48:20.758659 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:20.758049 6960 patch_prober.go:29] interesting pod/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-vjb5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 18:48:20.758659 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:20.758107 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-vjb5b" podUID="160390da-b35c-4030-9fff-d93afbcf9bad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 18:48:21.181187 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:21.181120 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:21.401838 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:21.401797 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:21.768045 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:21.766650 6960 patch_prober.go:29] interesting pod/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-vjb5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 18:48:21.768045 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:21.766715 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-vjb5b" podUID="160390da-b35c-4030-9fff-d93afbcf9bad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 18:48:21.912879 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:21.912702 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:22.236888 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:22.236851 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:22.439742 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:22.439711 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:22.643136 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:22.643095 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:22.758513 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:22.758481 6960 patch_prober.go:29] interesting pod/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-vjb5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 18:48:22.758805 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:22.758775 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-vjb5b" podUID="160390da-b35c-4030-9fff-d93afbcf9bad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 18:48:23.085778 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:23.085731 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:23.766164 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:23.758528 6960 patch_prober.go:29] interesting pod/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-vjb5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 18:48:23.766164 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:23.758628 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-vjb5b" podUID="160390da-b35c-4030-9fff-d93afbcf9bad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 18:48:23.876168 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:23.866688 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:23.932993 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:48:23.932960 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 18:48:23.980570 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:23.980534 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2aada387-558c97f7cf-5tzj8"] Oct 30 18:48:23.985407 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:23.985388 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2aada387-558c97f7cf-5tzj8" Oct 30 18:48:24.002515 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:24.002494 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-2aada387\"" Oct 30 18:48:24.006304 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:24.006289 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-2aada387-dockercfg-w6jt6\"" Oct 30 18:48:24.016407 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:24.016361 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2aada387-6db6c946b-g28xh"] Oct 30 18:48:24.020070 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:24.020055 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2aada387-6db6c946b-g28xh" Oct 30 18:48:24.028447 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:24.028413 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71cf7e67-8071-4da9-897c-d791c6b93fbc-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-2aada387-558c97f7cf-5tzj8\" (UID: \"71cf7e67-8071-4da9-897c-d791c6b93fbc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2aada387-558c97f7cf-5tzj8" Oct 30 18:48:24.028770 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:24.028744 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvb86\" (UniqueName: \"kubernetes.io/projected/71cf7e67-8071-4da9-897c-d791c6b93fbc-kube-api-access-wvb86\") pod \"payload-pre-processing-e2e-mt-2aada387-558c97f7cf-5tzj8\" (UID: \"71cf7e67-8071-4da9-897c-d791c6b93fbc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2aada387-558c97f7cf-5tzj8" Oct 30 18:48:24.081172 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:24.078248 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2aada387-558c97f7cf-5tzj8"] Oct 30 18:48:24.092123 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:24.092088 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2aada387-6db6c946b-g28xh"] Oct 30 18:48:24.127836 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:24.127800 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:24.133164 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:24.130053 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvb86\" (UniqueName: \"kubernetes.io/projected/71cf7e67-8071-4da9-897c-d791c6b93fbc-kube-api-access-wvb86\") pod \"payload-pre-processing-e2e-mt-2aada387-558c97f7cf-5tzj8\" (UID: \"71cf7e67-8071-4da9-897c-d791c6b93fbc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2aada387-558c97f7cf-5tzj8" Oct 30 18:48:24.133164 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:24.130134 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgvpj\" (UniqueName: \"kubernetes.io/projected/4a7b452b-b870-4f4b-97af-8bda0657c002-kube-api-access-lgvpj\") pod \"payload-processing-e2e-mt-2aada387-6db6c946b-g28xh\" (UID: \"4a7b452b-b870-4f4b-97af-8bda0657c002\") " pod="openshift-ingress/payload-processing-e2e-mt-2aada387-6db6c946b-g28xh" Oct 30 18:48:24.133164 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:24.130442 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71cf7e67-8071-4da9-897c-d791c6b93fbc-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-2aada387-558c97f7cf-5tzj8\" (UID: \"71cf7e67-8071-4da9-897c-d791c6b93fbc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2aada387-558c97f7cf-5tzj8" Oct 30 18:48:24.133164 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:24.130514 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a7b452b-b870-4f4b-97af-8bda0657c002-plugins-config-volume\") pod \"payload-processing-e2e-mt-2aada387-6db6c946b-g28xh\" (UID: \"4a7b452b-b870-4f4b-97af-8bda0657c002\") " pod="openshift-ingress/payload-processing-e2e-mt-2aada387-6db6c946b-g28xh" Oct 30 18:48:24.133164 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:24.132249 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71cf7e67-8071-4da9-897c-d791c6b93fbc-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-2aada387-558c97f7cf-5tzj8\" (UID: \"71cf7e67-8071-4da9-897c-d791c6b93fbc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2aada387-558c97f7cf-5tzj8" Oct 30 18:48:24.198637 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:24.198604 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvb86\" (UniqueName: \"kubernetes.io/projected/71cf7e67-8071-4da9-897c-d791c6b93fbc-kube-api-access-wvb86\") pod \"payload-pre-processing-e2e-mt-2aada387-558c97f7cf-5tzj8\" (UID: \"71cf7e67-8071-4da9-897c-d791c6b93fbc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2aada387-558c97f7cf-5tzj8" Oct 30 18:48:24.231870 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:24.231831 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a7b452b-b870-4f4b-97af-8bda0657c002-plugins-config-volume\") pod \"payload-processing-e2e-mt-2aada387-6db6c946b-g28xh\" (UID: \"4a7b452b-b870-4f4b-97af-8bda0657c002\") " pod="openshift-ingress/payload-processing-e2e-mt-2aada387-6db6c946b-g28xh" Oct 30 18:48:24.232487 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:24.232456 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgvpj\" (UniqueName: \"kubernetes.io/projected/4a7b452b-b870-4f4b-97af-8bda0657c002-kube-api-access-lgvpj\") pod \"payload-processing-e2e-mt-2aada387-6db6c946b-g28xh\" (UID: \"4a7b452b-b870-4f4b-97af-8bda0657c002\") " pod="openshift-ingress/payload-processing-e2e-mt-2aada387-6db6c946b-g28xh" Oct 30 18:48:24.232897 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:24.232871 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a7b452b-b870-4f4b-97af-8bda0657c002-plugins-config-volume\") pod \"payload-processing-e2e-mt-2aada387-6db6c946b-g28xh\" (UID: \"4a7b452b-b870-4f4b-97af-8bda0657c002\") " pod="openshift-ingress/payload-processing-e2e-mt-2aada387-6db6c946b-g28xh" Oct 30 18:48:24.248923 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:24.248901 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgvpj\" (UniqueName: \"kubernetes.io/projected/4a7b452b-b870-4f4b-97af-8bda0657c002-kube-api-access-lgvpj\") pod \"payload-processing-e2e-mt-2aada387-6db6c946b-g28xh\" (UID: \"4a7b452b-b870-4f4b-97af-8bda0657c002\") " pod="openshift-ingress/payload-processing-e2e-mt-2aada387-6db6c946b-g28xh" Oct 30 18:48:24.301239 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:24.300118 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2aada387-558c97f7cf-5tzj8" Oct 30 18:48:24.349651 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:24.349620 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2aada387-6db6c946b-g28xh" Oct 30 18:48:24.612353 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:48:24.612328 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71cf7e67_8071_4da9_897c_d791c6b93fbc.slice/crio-4945373a9bb26add8abdfc9da0cf844328d990a6135f60257ee6cdbfd4047142 WatchSource:0}: Error finding container 4945373a9bb26add8abdfc9da0cf844328d990a6135f60257ee6cdbfd4047142: Status 404 returned error can't find the container with id 4945373a9bb26add8abdfc9da0cf844328d990a6135f60257ee6cdbfd4047142 Oct 30 18:48:24.612732 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:24.612711 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2aada387-558c97f7cf-5tzj8"] Oct 30 18:48:24.660804 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:24.660780 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2aada387-6db6c946b-g28xh"] Oct 30 18:48:24.662303 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:48:24.661714 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a7b452b_b870_4f4b_97af_8bda0657c002.slice/crio-82794cdc58681006ed67c3074462e05015400a44964138af3cef67c3a475924a WatchSource:0}: Error finding container 82794cdc58681006ed67c3074462e05015400a44964138af3cef67c3a475924a: Status 404 returned error can't find the container with id 82794cdc58681006ed67c3074462e05015400a44964138af3cef67c3a475924a Oct 30 18:48:24.700670 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:24.700629 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2aada387-558c97f7cf-5tzj8" event={"ID":"71cf7e67-8071-4da9-897c-d791c6b93fbc","Type":"ContainerStarted","Data":"a0eac54c9c25956a0ddeb8647ca2811a8bb68cf6a69cd19f54e7e4c93f4f282d"} Oct 30 18:48:24.700829 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:24.700700 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2aada387-558c97f7cf-5tzj8" event={"ID":"71cf7e67-8071-4da9-897c-d791c6b93fbc","Type":"ContainerStarted","Data":"4945373a9bb26add8abdfc9da0cf844328d990a6135f60257ee6cdbfd4047142"} Oct 30 18:48:24.701472 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:24.701442 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-2aada387-558c97f7cf-5tzj8" Oct 30 18:48:24.702999 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:24.702969 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2aada387-6db6c946b-g28xh" event={"ID":"4a7b452b-b870-4f4b-97af-8bda0657c002","Type":"ContainerStarted","Data":"82794cdc58681006ed67c3074462e05015400a44964138af3cef67c3a475924a"} Oct 30 18:48:24.728169 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:24.728075 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-2aada387-558c97f7cf-5tzj8" podStartSLOduration=1.72805649 podStartE2EDuration="1.72805649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:48:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:48:24.723293238 +0000 UTC m=+872.252324377" watchObservedRunningTime="2026-10-30 18:48:24.72805649 +0000 UTC m=+872.257087629" Oct 30 18:48:24.758566 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:24.758538 6960 patch_prober.go:29] interesting pod/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-vjb5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 18:48:24.758700 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:24.758592 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-vjb5b" podUID="160390da-b35c-4030-9fff-d93afbcf9bad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 18:48:24.772083 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:24.772057 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:25.022124 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:25.022039 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-xg72j"] Oct 30 18:48:25.026246 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:25.026226 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-xg72j" Oct 30 18:48:25.042693 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:25.042667 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-xg72j"] Oct 30 18:48:25.043469 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:25.043443 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4522747f-f5bf-481e-a35c-58e3475a3221-workload-certs\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-xg72j\" (UID: \"4522747f-f5bf-481e-a35c-58e3475a3221\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-xg72j" Oct 30 18:48:25.043565 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:25.043522 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4522747f-f5bf-481e-a35c-58e3475a3221-istio-podinfo\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-xg72j\" (UID: \"4522747f-f5bf-481e-a35c-58e3475a3221\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-xg72j" Oct 30 18:48:25.043697 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:25.043676 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4522747f-f5bf-481e-a35c-58e3475a3221-istio-envoy\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-xg72j\" (UID: \"4522747f-f5bf-481e-a35c-58e3475a3221\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-xg72j" Oct 30 18:48:25.043785 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:25.043768 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4522747f-f5bf-481e-a35c-58e3475a3221-istio-token\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-xg72j\" (UID: \"4522747f-f5bf-481e-a35c-58e3475a3221\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-xg72j" Oct 30 18:48:25.043844 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:25.043803 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4522747f-f5bf-481e-a35c-58e3475a3221-credential-socket\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-xg72j\" (UID: \"4522747f-f5bf-481e-a35c-58e3475a3221\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-xg72j" Oct 30 18:48:25.043844 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:25.043828 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4522747f-f5bf-481e-a35c-58e3475a3221-istiod-ca-cert\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-xg72j\" (UID: \"4522747f-f5bf-481e-a35c-58e3475a3221\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-xg72j" Oct 30 18:48:25.044020 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:25.044004 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4522747f-f5bf-481e-a35c-58e3475a3221-workload-socket\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-xg72j\" (UID: \"4522747f-f5bf-481e-a35c-58e3475a3221\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-xg72j" Oct 30 18:48:25.044086 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:25.044049 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4522747f-f5bf-481e-a35c-58e3475a3221-istio-data\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-xg72j\" (UID: \"4522747f-f5bf-481e-a35c-58e3475a3221\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-xg72j" Oct 30 18:48:25.044086 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:25.044078 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r8sh\" (UniqueName: \"kubernetes.io/projected/4522747f-f5bf-481e-a35c-58e3475a3221-kube-api-access-9r8sh\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-xg72j\" (UID: \"4522747f-f5bf-481e-a35c-58e3475a3221\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-xg72j" Oct 30 18:48:25.145006 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:25.144975 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4522747f-f5bf-481e-a35c-58e3475a3221-workload-certs\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-xg72j\" (UID: \"4522747f-f5bf-481e-a35c-58e3475a3221\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-xg72j" Oct 30 18:48:25.145217 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:25.145028 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4522747f-f5bf-481e-a35c-58e3475a3221-istio-podinfo\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-xg72j\" (UID: \"4522747f-f5bf-481e-a35c-58e3475a3221\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-xg72j" Oct 30 18:48:25.145217 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:25.145054 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4522747f-f5bf-481e-a35c-58e3475a3221-istio-envoy\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-xg72j\" (UID: \"4522747f-f5bf-481e-a35c-58e3475a3221\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-xg72j" Oct 30 18:48:25.145217 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:25.145085 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4522747f-f5bf-481e-a35c-58e3475a3221-istio-token\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-xg72j\" (UID: \"4522747f-f5bf-481e-a35c-58e3475a3221\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-xg72j" Oct 30 18:48:25.145217 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:25.145105 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4522747f-f5bf-481e-a35c-58e3475a3221-credential-socket\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-xg72j\" (UID: \"4522747f-f5bf-481e-a35c-58e3475a3221\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-xg72j" Oct 30 18:48:25.145217 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:25.145120 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4522747f-f5bf-481e-a35c-58e3475a3221-istiod-ca-cert\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-xg72j\" (UID: \"4522747f-f5bf-481e-a35c-58e3475a3221\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-xg72j" Oct 30 18:48:25.145522 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:25.145254 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4522747f-f5bf-481e-a35c-58e3475a3221-workload-socket\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-xg72j\" (UID: \"4522747f-f5bf-481e-a35c-58e3475a3221\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-xg72j" Oct 30 18:48:25.145522 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:25.145294 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4522747f-f5bf-481e-a35c-58e3475a3221-istio-data\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-xg72j\" (UID: \"4522747f-f5bf-481e-a35c-58e3475a3221\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-xg72j" Oct 30 18:48:25.145522 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:25.145312 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r8sh\" (UniqueName: \"kubernetes.io/projected/4522747f-f5bf-481e-a35c-58e3475a3221-kube-api-access-9r8sh\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-xg72j\" (UID: \"4522747f-f5bf-481e-a35c-58e3475a3221\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-xg72j" Oct 30 18:48:25.145522 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:25.145454 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4522747f-f5bf-481e-a35c-58e3475a3221-workload-certs\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-xg72j\" (UID: \"4522747f-f5bf-481e-a35c-58e3475a3221\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-xg72j" Oct 30 18:48:25.145746 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:25.145582 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4522747f-f5bf-481e-a35c-58e3475a3221-credential-socket\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-xg72j\" (UID: \"4522747f-f5bf-481e-a35c-58e3475a3221\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-xg72j" Oct 30 18:48:25.145746 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:25.145734 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4522747f-f5bf-481e-a35c-58e3475a3221-istio-data\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-xg72j\" (UID: \"4522747f-f5bf-481e-a35c-58e3475a3221\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-xg72j" Oct 30 18:48:25.145981 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:25.145813 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4522747f-f5bf-481e-a35c-58e3475a3221-workload-socket\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-xg72j\" (UID: \"4522747f-f5bf-481e-a35c-58e3475a3221\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-xg72j" Oct 30 18:48:25.146284 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:25.146256 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4522747f-f5bf-481e-a35c-58e3475a3221-istiod-ca-cert\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-xg72j\" (UID: \"4522747f-f5bf-481e-a35c-58e3475a3221\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-xg72j" Oct 30 18:48:25.147617 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:25.147597 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4522747f-f5bf-481e-a35c-58e3475a3221-istio-envoy\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-xg72j\" (UID: \"4522747f-f5bf-481e-a35c-58e3475a3221\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-xg72j" Oct 30 18:48:25.147700 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:25.147681 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4522747f-f5bf-481e-a35c-58e3475a3221-istio-podinfo\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-xg72j\" (UID: \"4522747f-f5bf-481e-a35c-58e3475a3221\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-xg72j" Oct 30 18:48:25.158094 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:25.158004 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4522747f-f5bf-481e-a35c-58e3475a3221-istio-token\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-xg72j\" (UID: \"4522747f-f5bf-481e-a35c-58e3475a3221\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-xg72j" Oct 30 18:48:25.159602 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:25.159583 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r8sh\" (UniqueName: \"kubernetes.io/projected/4522747f-f5bf-481e-a35c-58e3475a3221-kube-api-access-9r8sh\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-xg72j\" (UID: \"4522747f-f5bf-481e-a35c-58e3475a3221\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-xg72j" Oct 30 18:48:25.341900 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:25.341868 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-xg72j" Oct 30 18:48:25.662035 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:25.661962 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-xg72j"] Oct 30 18:48:25.666753 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:48:25.666726 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4522747f_f5bf_481e_a35c_58e3475a3221.slice/crio-6d9f4a95e978dc48d54c8896f687d2be82479c246a4fc09669b37d6179747aa0 WatchSource:0}: Error finding container 6d9f4a95e978dc48d54c8896f687d2be82479c246a4fc09669b37d6179747aa0: Status 404 returned error can't find the container with id 6d9f4a95e978dc48d54c8896f687d2be82479c246a4fc09669b37d6179747aa0 Oct 30 18:48:25.669544 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:25.669507 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:48:25.669644 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:25.669592 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:48:25.669644 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:25.669636 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:48:25.712478 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:25.712442 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2aada387-6db6c946b-g28xh" event={"ID":"4a7b452b-b870-4f4b-97af-8bda0657c002","Type":"ContainerStarted","Data":"98e0b39069928ca2a2891d69be94cb6a9fc54fb242f4bd2a5804ca4744f54797"} Oct 30 18:48:25.712607 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:25.712516 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-2aada387-6db6c946b-g28xh" Oct 30 18:48:25.714694 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:25.714666 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-xg72j" event={"ID":"4522747f-f5bf-481e-a35c-58e3475a3221","Type":"ContainerStarted","Data":"6d9f4a95e978dc48d54c8896f687d2be82479c246a4fc09669b37d6179747aa0"} Oct 30 18:48:25.746702 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:25.744014 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-2aada387-6db6c946b-g28xh" podStartSLOduration=2.7439945100000003 podStartE2EDuration="2.74399451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:48:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:48:25.741559074 +0000 UTC m=+873.270590212" watchObservedRunningTime="2026-10-30 18:48:25.74399451 +0000 UTC m=+873.273025651" Oct 30 18:48:25.820059 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:25.820029 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-vjb5b" Oct 30 18:48:25.821716 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:25.821631 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-vjb5b" Oct 30 18:48:25.822462 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:25.822413 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-vjb5b" Oct 30 18:48:26.134882 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:26.134843 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:26.441532 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:26.441441 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:26.722607 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:26.722517 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-xg72j" event={"ID":"4522747f-f5bf-481e-a35c-58e3475a3221","Type":"ContainerStarted","Data":"0e884b87ae4850a10a689779c11c2f61787d9364c30ed5bd1157ee399ce63096"} Oct 30 18:48:27.075169 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:27.074453 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:27.284487 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:27.284450 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:27.343045 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:27.342963 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-xg72j" Oct 30 18:48:27.344789 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:27.344761 6960 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-xg72j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 18:48:27.344936 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:27.344817 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-xg72j" podUID="4522747f-f5bf-481e-a35c-58e3475a3221" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 18:48:27.484117 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:27.484075 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:27.632725 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:27.632616 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:28.012817 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:28.012697 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:28.343170 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:28.343107 6960 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-xg72j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 18:48:28.343561 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:28.343191 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-xg72j" podUID="4522747f-f5bf-481e-a35c-58e3475a3221" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 18:48:28.949689 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:28.949656 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:29.342627 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:29.342593 6960 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-xg72j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 18:48:29.342806 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:29.342645 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-xg72j" podUID="4522747f-f5bf-481e-a35c-58e3475a3221" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 18:48:29.868170 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:29.863652 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-xg72j" podStartSLOduration=5.863631475 podStartE2EDuration="5.863631475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:48:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:48:26.781094993 +0000 UTC m=+874.310126135" watchObservedRunningTime="2026-10-30 18:48:29.863631475 +0000 UTC m=+877.392662616" Oct 30 18:48:29.872416 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:29.872384 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2aada387-577477546b-4fdz6"] Oct 30 18:48:29.898404 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:29.898216 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2aada387-577477546b-4fdz6" Oct 30 18:48:29.902661 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:29.900214 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2aada387-577477546b-4fdz6"] Oct 30 18:48:29.910780 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:29.910588 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-2aada387-serving-cert\"" Oct 30 18:48:29.943770 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:29.943745 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dcdd0a5f-02b9-41be-b090-aa2468942295-maas-api-tls\") pod \"maas-api-e2e-mt-2aada387-577477546b-4fdz6\" (UID: \"dcdd0a5f-02b9-41be-b090-aa2468942295\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2aada387-577477546b-4fdz6" Oct 30 18:48:29.943887 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:29.943823 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkrzx\" (UniqueName: \"kubernetes.io/projected/dcdd0a5f-02b9-41be-b090-aa2468942295-kube-api-access-vkrzx\") pod \"maas-api-e2e-mt-2aada387-577477546b-4fdz6\" (UID: \"dcdd0a5f-02b9-41be-b090-aa2468942295\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2aada387-577477546b-4fdz6" Oct 30 18:48:29.943887 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:29.943850 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcdd0a5f-02b9-41be-b090-aa2468942295-metrics-certs\") pod \"maas-api-e2e-mt-2aada387-577477546b-4fdz6\" (UID: \"dcdd0a5f-02b9-41be-b090-aa2468942295\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2aada387-577477546b-4fdz6" Oct 30 18:48:29.943972 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:29.943950 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcdd0a5f-02b9-41be-b090-aa2468942295-openshift-service-ca\") pod \"maas-api-e2e-mt-2aada387-577477546b-4fdz6\" (UID: \"dcdd0a5f-02b9-41be-b090-aa2468942295\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2aada387-577477546b-4fdz6" Oct 30 18:48:29.980762 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:29.980719 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:30.045619 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:30.045580 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcdd0a5f-02b9-41be-b090-aa2468942295-openshift-service-ca\") pod \"maas-api-e2e-mt-2aada387-577477546b-4fdz6\" (UID: \"dcdd0a5f-02b9-41be-b090-aa2468942295\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2aada387-577477546b-4fdz6" Oct 30 18:48:30.045791 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:30.045722 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dcdd0a5f-02b9-41be-b090-aa2468942295-maas-api-tls\") pod \"maas-api-e2e-mt-2aada387-577477546b-4fdz6\" (UID: \"dcdd0a5f-02b9-41be-b090-aa2468942295\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2aada387-577477546b-4fdz6" Oct 30 18:48:30.045873 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:30.045797 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkrzx\" (UniqueName: \"kubernetes.io/projected/dcdd0a5f-02b9-41be-b090-aa2468942295-kube-api-access-vkrzx\") pod \"maas-api-e2e-mt-2aada387-577477546b-4fdz6\" (UID: \"dcdd0a5f-02b9-41be-b090-aa2468942295\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2aada387-577477546b-4fdz6" Oct 30 18:48:30.045873 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:30.045815 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcdd0a5f-02b9-41be-b090-aa2468942295-metrics-certs\") pod \"maas-api-e2e-mt-2aada387-577477546b-4fdz6\" (UID: \"dcdd0a5f-02b9-41be-b090-aa2468942295\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2aada387-577477546b-4fdz6" Oct 30 18:48:30.045873 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:48:30.045833 6960 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-2aada387-serving-cert: secret "maas-api-e2e-mt-2aada387-serving-cert" not found Oct 30 18:48:30.046025 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:48:30.045926 6960 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcdd0a5f-02b9-41be-b090-aa2468942295-maas-api-tls podName:dcdd0a5f-02b9-41be-b090-aa2468942295 nodeName:}" failed. No retries permitted until 2026-10-30 18:48:30.545905851 +0000 UTC m=+878.074936988 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/dcdd0a5f-02b9-41be-b090-aa2468942295-maas-api-tls") pod "maas-api-e2e-mt-2aada387-577477546b-4fdz6" (UID: "dcdd0a5f-02b9-41be-b090-aa2468942295") : secret "maas-api-e2e-mt-2aada387-serving-cert" not found Oct 30 18:48:30.046325 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:30.046303 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcdd0a5f-02b9-41be-b090-aa2468942295-openshift-service-ca\") pod \"maas-api-e2e-mt-2aada387-577477546b-4fdz6\" (UID: \"dcdd0a5f-02b9-41be-b090-aa2468942295\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2aada387-577477546b-4fdz6" Oct 30 18:48:30.048559 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:30.048539 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcdd0a5f-02b9-41be-b090-aa2468942295-metrics-certs\") pod \"maas-api-e2e-mt-2aada387-577477546b-4fdz6\" (UID: \"dcdd0a5f-02b9-41be-b090-aa2468942295\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2aada387-577477546b-4fdz6" Oct 30 18:48:30.084168 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:30.082641 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkrzx\" (UniqueName: \"kubernetes.io/projected/dcdd0a5f-02b9-41be-b090-aa2468942295-kube-api-access-vkrzx\") pod \"maas-api-e2e-mt-2aada387-577477546b-4fdz6\" (UID: \"dcdd0a5f-02b9-41be-b090-aa2468942295\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2aada387-577477546b-4fdz6" Oct 30 18:48:30.093713 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:30.093668 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:30.204067 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:30.203969 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:30.342533 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:30.342495 6960 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-xg72j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 18:48:30.342704 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:30.342559 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-xg72j" podUID="4522747f-f5bf-481e-a35c-58e3475a3221" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 18:48:30.565168 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:30.555434 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dcdd0a5f-02b9-41be-b090-aa2468942295-maas-api-tls\") pod \"maas-api-e2e-mt-2aada387-577477546b-4fdz6\" (UID: \"dcdd0a5f-02b9-41be-b090-aa2468942295\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2aada387-577477546b-4fdz6" Oct 30 18:48:30.569169 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:30.565494 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dcdd0a5f-02b9-41be-b090-aa2468942295-maas-api-tls\") pod \"maas-api-e2e-mt-2aada387-577477546b-4fdz6\" (UID: \"dcdd0a5f-02b9-41be-b090-aa2468942295\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2aada387-577477546b-4fdz6" Oct 30 18:48:30.821360 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:30.821275 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2aada387-577477546b-4fdz6" Oct 30 18:48:31.128989 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:31.128909 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2aada387-577477546b-4fdz6"] Oct 30 18:48:31.131578 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:31.131550 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:31.134239 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:48:31.134210 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcdd0a5f_02b9_41be_b090_aa2468942295.slice/crio-37c428beba8a1cbb69fb8c96a88cbc1d98ae27d329013f87762b1f8f2e9dfc29 WatchSource:0}: Error finding container 37c428beba8a1cbb69fb8c96a88cbc1d98ae27d329013f87762b1f8f2e9dfc29: Status 404 returned error can't find the container with id 37c428beba8a1cbb69fb8c96a88cbc1d98ae27d329013f87762b1f8f2e9dfc29 Oct 30 18:48:31.343225 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:31.343192 6960 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-xg72j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 18:48:31.343401 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:31.343244 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-xg72j" podUID="4522747f-f5bf-481e-a35c-58e3475a3221" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 18:48:31.810979 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:31.810916 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2aada387-577477546b-4fdz6" event={"ID":"dcdd0a5f-02b9-41be-b090-aa2468942295","Type":"ContainerStarted","Data":"c3beaf7efb52400028f8c4e06f44e5f18ed430dab6359d9c85d08f1e6cc07307"} Oct 30 18:48:31.811174 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:31.810995 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2aada387-577477546b-4fdz6" event={"ID":"dcdd0a5f-02b9-41be-b090-aa2468942295","Type":"ContainerStarted","Data":"37c428beba8a1cbb69fb8c96a88cbc1d98ae27d329013f87762b1f8f2e9dfc29"} Oct 30 18:48:31.811174 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:31.811123 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2aada387-577477546b-4fdz6" Oct 30 18:48:31.833280 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:31.833227 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2aada387-577477546b-4fdz6" podStartSLOduration=2.321681549 podStartE2EDuration="2.833212916s" totalImagesPullingTime="511.531367ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:48:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:48:31.830499037 +0000 UTC m=+879.359530202" watchObservedRunningTime="2026-10-30 18:48:31.833212916 +0000 UTC m=+879.362244093" Oct 30 18:48:32.006644 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:32.006538 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:32.343242 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:32.343208 6960 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-xg72j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 18:48:32.343673 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:32.343259 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-xg72j" podUID="4522747f-f5bf-481e-a35c-58e3475a3221" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 18:48:32.384387 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:32.384351 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-frrjm"] Oct 30 18:48:32.391460 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:32.391429 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-frrjm" Oct 30 18:48:32.420530 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:32.420503 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-frrjm"] Oct 30 18:48:32.521606 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:32.521565 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p2bm\" (UniqueName: \"kubernetes.io/projected/a5cb9eed-a657-4907-9765-9114f12fb473-kube-api-access-5p2bm\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-frrjm\" (UID: \"a5cb9eed-a657-4907-9765-9114f12fb473\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-frrjm" Oct 30 18:48:32.521778 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:32.521625 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5cb9eed-a657-4907-9765-9114f12fb473-workload-socket\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-frrjm\" (UID: \"a5cb9eed-a657-4907-9765-9114f12fb473\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-frrjm" Oct 30 18:48:32.521778 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:32.521718 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5cb9eed-a657-4907-9765-9114f12fb473-workload-certs\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-frrjm\" (UID: \"a5cb9eed-a657-4907-9765-9114f12fb473\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-frrjm" Oct 30 18:48:32.521778 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:32.521745 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5cb9eed-a657-4907-9765-9114f12fb473-istio-data\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-frrjm\" (UID: \"a5cb9eed-a657-4907-9765-9114f12fb473\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-frrjm" Oct 30 18:48:32.521950 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:32.521860 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5cb9eed-a657-4907-9765-9114f12fb473-credential-socket\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-frrjm\" (UID: \"a5cb9eed-a657-4907-9765-9114f12fb473\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-frrjm" Oct 30 18:48:32.521950 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:32.521886 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5cb9eed-a657-4907-9765-9114f12fb473-istio-envoy\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-frrjm\" (UID: \"a5cb9eed-a657-4907-9765-9114f12fb473\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-frrjm" Oct 30 18:48:32.522024 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:32.522010 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5cb9eed-a657-4907-9765-9114f12fb473-istio-podinfo\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-frrjm\" (UID: \"a5cb9eed-a657-4907-9765-9114f12fb473\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-frrjm" Oct 30 18:48:32.522073 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:32.522060 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5cb9eed-a657-4907-9765-9114f12fb473-istio-token\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-frrjm\" (UID: \"a5cb9eed-a657-4907-9765-9114f12fb473\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-frrjm" Oct 30 18:48:32.522125 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:32.522081 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5cb9eed-a657-4907-9765-9114f12fb473-istiod-ca-cert\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-frrjm\" (UID: \"a5cb9eed-a657-4907-9765-9114f12fb473\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-frrjm" Oct 30 18:48:32.623376 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:32.623277 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5cb9eed-a657-4907-9765-9114f12fb473-istio-token\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-frrjm\" (UID: \"a5cb9eed-a657-4907-9765-9114f12fb473\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-frrjm" Oct 30 18:48:32.623376 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:32.623327 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5cb9eed-a657-4907-9765-9114f12fb473-istiod-ca-cert\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-frrjm\" (UID: \"a5cb9eed-a657-4907-9765-9114f12fb473\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-frrjm" Oct 30 18:48:32.623607 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:32.623481 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p2bm\" (UniqueName: \"kubernetes.io/projected/a5cb9eed-a657-4907-9765-9114f12fb473-kube-api-access-5p2bm\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-frrjm\" (UID: \"a5cb9eed-a657-4907-9765-9114f12fb473\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-frrjm" Oct 30 18:48:32.623607 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:32.623545 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5cb9eed-a657-4907-9765-9114f12fb473-workload-socket\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-frrjm\" (UID: \"a5cb9eed-a657-4907-9765-9114f12fb473\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-frrjm" Oct 30 18:48:32.623607 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:32.623586 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5cb9eed-a657-4907-9765-9114f12fb473-workload-certs\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-frrjm\" (UID: \"a5cb9eed-a657-4907-9765-9114f12fb473\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-frrjm" Oct 30 18:48:32.623780 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:32.623615 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5cb9eed-a657-4907-9765-9114f12fb473-istio-data\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-frrjm\" (UID: \"a5cb9eed-a657-4907-9765-9114f12fb473\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-frrjm" Oct 30 18:48:32.623780 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:32.623675 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5cb9eed-a657-4907-9765-9114f12fb473-credential-socket\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-frrjm\" (UID: \"a5cb9eed-a657-4907-9765-9114f12fb473\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-frrjm" Oct 30 18:48:32.623780 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:32.623702 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5cb9eed-a657-4907-9765-9114f12fb473-istio-envoy\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-frrjm\" (UID: \"a5cb9eed-a657-4907-9765-9114f12fb473\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-frrjm" Oct 30 18:48:32.623955 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:32.623855 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5cb9eed-a657-4907-9765-9114f12fb473-istio-podinfo\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-frrjm\" (UID: \"a5cb9eed-a657-4907-9765-9114f12fb473\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-frrjm" Oct 30 18:48:32.624015 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:32.623989 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5cb9eed-a657-4907-9765-9114f12fb473-workload-socket\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-frrjm\" (UID: \"a5cb9eed-a657-4907-9765-9114f12fb473\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-frrjm" Oct 30 18:48:32.624115 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:32.624076 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5cb9eed-a657-4907-9765-9114f12fb473-istiod-ca-cert\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-frrjm\" (UID: \"a5cb9eed-a657-4907-9765-9114f12fb473\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-frrjm" Oct 30 18:48:32.624208 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:32.624117 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5cb9eed-a657-4907-9765-9114f12fb473-istio-data\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-frrjm\" (UID: \"a5cb9eed-a657-4907-9765-9114f12fb473\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-frrjm" Oct 30 18:48:32.624279 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:32.624259 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5cb9eed-a657-4907-9765-9114f12fb473-workload-certs\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-frrjm\" (UID: \"a5cb9eed-a657-4907-9765-9114f12fb473\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-frrjm" Oct 30 18:48:32.624571 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:32.624552 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5cb9eed-a657-4907-9765-9114f12fb473-credential-socket\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-frrjm\" (UID: \"a5cb9eed-a657-4907-9765-9114f12fb473\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-frrjm" Oct 30 18:48:32.626230 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:32.626208 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5cb9eed-a657-4907-9765-9114f12fb473-istio-envoy\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-frrjm\" (UID: \"a5cb9eed-a657-4907-9765-9114f12fb473\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-frrjm" Oct 30 18:48:32.626587 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:32.626566 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5cb9eed-a657-4907-9765-9114f12fb473-istio-podinfo\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-frrjm\" (UID: \"a5cb9eed-a657-4907-9765-9114f12fb473\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-frrjm" Oct 30 18:48:32.636433 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:32.636412 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5cb9eed-a657-4907-9765-9114f12fb473-istio-token\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-frrjm\" (UID: \"a5cb9eed-a657-4907-9765-9114f12fb473\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-frrjm" Oct 30 18:48:32.636682 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:32.636665 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p2bm\" (UniqueName: \"kubernetes.io/projected/a5cb9eed-a657-4907-9765-9114f12fb473-kube-api-access-5p2bm\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-frrjm\" (UID: \"a5cb9eed-a657-4907-9765-9114f12fb473\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-frrjm" Oct 30 18:48:32.709723 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:32.709678 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-frrjm" Oct 30 18:48:32.992751 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:32.992673 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:33.041172 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:33.037301 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-frrjm"] Oct 30 18:48:33.045619 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:33.045251 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:48:33.045619 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:33.045349 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:48:33.045619 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:33.045393 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:48:33.346734 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:33.346708 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-xg72j" Oct 30 18:48:33.347179 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:33.347051 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-xg72j" Oct 30 18:48:33.347902 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:33.347883 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-xg72j" Oct 30 18:48:33.829535 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:33.829494 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-frrjm" event={"ID":"a5cb9eed-a657-4907-9765-9114f12fb473","Type":"ContainerStarted","Data":"4b5fc94da92519effb48045968ef28abe79fa131cd16f3f914cac50f186e7cc5"} Oct 30 18:48:33.829710 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:33.829579 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-frrjm" event={"ID":"a5cb9eed-a657-4907-9765-9114f12fb473","Type":"ContainerStarted","Data":"5c6ab46f016829d5812e3a061b3433633607b298d86360c904dad6254cffeff7"} Oct 30 18:48:33.857349 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:33.857300 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-frrjm" podStartSLOduration=1.857285743 podStartE2EDuration="1.857285743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:48:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:48:33.855243597 +0000 UTC m=+881.384274734" watchObservedRunningTime="2026-10-30 18:48:33.857285743 +0000 UTC m=+881.386316881" Oct 30 18:48:34.291967 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:34.291888 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:34.448340 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:34.448304 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:34.617880 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:34.617844 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:34.710869 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:34.710823 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-frrjm" Oct 30 18:48:34.712684 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:34.712651 6960 patch_prober.go:29] interesting pod/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-frrjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 18:48:34.713000 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:34.712727 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-frrjm" podUID="a5cb9eed-a657-4907-9765-9114f12fb473" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 18:48:34.901885 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:34.901793 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:35.152434 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:35.152344 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:35.700304 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:35.700277 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:35.711069 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:35.711040 6960 patch_prober.go:29] interesting pod/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-frrjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 18:48:35.711239 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:35.711086 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-frrjm" podUID="a5cb9eed-a657-4907-9765-9114f12fb473" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 18:48:36.634358 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:36.634320 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:36.710897 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:36.710866 6960 patch_prober.go:29] interesting pod/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-frrjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 18:48:36.711374 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:36.710918 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-frrjm" podUID="a5cb9eed-a657-4907-9765-9114f12fb473" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 18:48:36.726217 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:36.726193 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-2aada387-6db6c946b-g28xh" Oct 30 18:48:36.726350 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:36.726264 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-2aada387-558c97f7cf-5tzj8" Oct 30 18:48:37.710756 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:37.710715 6960 patch_prober.go:29] interesting pod/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-frrjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 18:48:37.710944 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:37.710790 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-frrjm" podUID="a5cb9eed-a657-4907-9765-9114f12fb473" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 18:48:37.728691 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:37.728653 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:37.829002 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:37.828969 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2aada387-577477546b-4fdz6" Oct 30 18:48:38.716817 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:38.716765 6960 patch_prober.go:29] interesting pod/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-frrjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 18:48:38.717484 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:38.717436 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-frrjm" podUID="a5cb9eed-a657-4907-9765-9114f12fb473" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 18:48:39.504292 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:39.503256 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7a8fa2/simulator-w3-7a8fa2-kserve-78587cb8fc-6pvsz" podUID="cc64ebb9-44e9-435a-9db0-d53ce5f139ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:48:39.551518 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:39.551462 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-2-w2-8a92cc-kserve-6cc4b9c974-n7cd2" podUID="e5fd8f33-8662-4f90-9817-e601f694bf1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 18:48:39.551883 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:39.551850 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/simulator-w2-8a92cc-kserve-fb77d84bc-nqf9q" podUID="ce70492e-c727-4dc1-b2ee-bf278d29a45b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 18:48:39.552629 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:39.552590 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-aa6736/premium-w1-aa6736-kserve-6f986b66f7-nkfrg" podUID="84ca40c5-7033-48c4-bcec-e9ac7c034a4e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:48:39.628619 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:39.628578 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-aa6736/distinct-w1-aa6736-kserve-b86d7667b-smcbz" podUID="58c23db7-11cb-4ce2-a67e-fe698e89c71d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:48:39.637650 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:39.637245 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-aa6736/simulator-w1-aa6736-kserve-76985dcd5b-njvnn" podUID="943f7840-8d54-4845-b7ec-c0da72ba9c67" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:48:39.729314 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:39.729277 6960 patch_prober.go:29] interesting pod/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-frrjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 18:48:39.749712 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:39.749188 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-frrjm" podUID="a5cb9eed-a657-4907-9765-9114f12fb473" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 18:48:39.749940 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:39.748247 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-2-w3-7a8fa2-kserve-677bb44d8d-fg55k" podUID="e8701242-b59c-43b1-9c0a-011635edd419" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 30 18:48:40.240827 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:40.240789 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:40.466173 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:40.462674 6960 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-aa6736/simulator-w1-aa6736-kserve-76985dcd5b-njvnn" podUID="943f7840-8d54-4845-b7ec-c0da72ba9c67" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": context deadline exceeded" Oct 30 18:48:40.466173 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:40.462995 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7a8fa2/premium-w3-7a8fa2-kserve-6b6c5f6fb8-n5wtq" podUID="f51fd746-cd85-4608-96a6-f54afa0ec34d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 30 18:48:40.512237 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:40.506245 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7a8fa2/simulator-w3-7a8fa2-kserve-78587cb8fc-6pvsz" podUID="cc64ebb9-44e9-435a-9db0-d53ce5f139ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:48:40.512237 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:40.506689 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7a8fa2/simulator-w3-7a8fa2-kserve-78587cb8fc-6pvsz" Oct 30 18:48:40.555439 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:40.555382 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-2-w2-8a92cc-kserve-6cc4b9c974-n7cd2" podUID="e5fd8f33-8662-4f90-9817-e601f694bf1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:48:40.560488 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:40.559671 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-aa6736/premium-w1-aa6736-kserve-6f986b66f7-nkfrg" podUID="84ca40c5-7033-48c4-bcec-e9ac7c034a4e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:48:40.582225 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:40.566301 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/simulator-w2-8a92cc-kserve-fb77d84bc-nqf9q" podUID="ce70492e-c727-4dc1-b2ee-bf278d29a45b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 18:48:40.582225 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:40.567058 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-2-w2-8a92cc-kserve-6cc4b9c974-n7cd2" Oct 30 18:48:40.582225 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:40.567137 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-aa6736/premium-w1-aa6736-kserve-6f986b66f7-nkfrg" Oct 30 18:48:40.582225 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:40.567198 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8a92cc/simulator-w2-8a92cc-kserve-fb77d84bc-nqf9q" Oct 30 18:48:40.648675 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:40.646428 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-aa6736/simulator-w1-aa6736-kserve-76985dcd5b-njvnn" podUID="943f7840-8d54-4845-b7ec-c0da72ba9c67" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:48:40.648675 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:40.647314 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-aa6736/distinct-w1-aa6736-kserve-b86d7667b-smcbz" podUID="58c23db7-11cb-4ce2-a67e-fe698e89c71d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 18:48:40.713611 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:40.713489 6960 patch_prober.go:29] interesting pod/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-frrjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 18:48:40.713611 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:40.713547 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-frrjm" podUID="a5cb9eed-a657-4907-9765-9114f12fb473" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 18:48:40.734927 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:40.734899 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-aa6736/simulator-w1-aa6736-kserve-76985dcd5b-njvnn" Oct 30 18:48:40.735567 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:40.735549 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-aa6736/distinct-w1-aa6736-kserve-b86d7667b-smcbz" Oct 30 18:48:40.772266 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:40.771917 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-2-w3-7a8fa2-kserve-677bb44d8d-fg55k" podUID="e8701242-b59c-43b1-9c0a-011635edd419" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 30 18:48:40.774332 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:40.772565 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-2-w3-7a8fa2-kserve-677bb44d8d-fg55k" Oct 30 18:48:40.805376 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:40.804834 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:40.947534 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:40.947505 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-2-w2-8a92cc-kserve-6cc4b9c974-n7cd2" Oct 30 18:48:40.980815 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:40.980783 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:41.148398 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:41.148350 6960 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-aa6736/distinct-w1-aa6736-kserve-b86d7667b-smcbz" podUID="58c23db7-11cb-4ce2-a67e-fe698e89c71d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:48:41.477226 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:41.470946 6960 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-8a92cc/simulator-w2-8a92cc-kserve-fb77d84bc-nqf9q" podUID="ce70492e-c727-4dc1-b2ee-bf278d29a45b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": context deadline exceeded" Oct 30 18:48:41.510626 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:41.509781 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7a8fa2/simulator-w3-7a8fa2-kserve-78587cb8fc-6pvsz" podUID="cc64ebb9-44e9-435a-9db0-d53ce5f139ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 30 18:48:41.523633 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:41.521335 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7a8fa2/premium-w3-7a8fa2-kserve-6b6c5f6fb8-n5wtq" podUID="f51fd746-cd85-4608-96a6-f54afa0ec34d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:48:41.523633 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:41.521755 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-7a8fa2/premium-w3-7a8fa2-kserve-6b6c5f6fb8-n5wtq" Oct 30 18:48:41.563299 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:41.563247 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-aa6736/premium-w1-aa6736-kserve-6f986b66f7-nkfrg" podUID="84ca40c5-7033-48c4-bcec-e9ac7c034a4e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:48:41.581670 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:41.581628 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/simulator-w2-8a92cc-kserve-fb77d84bc-nqf9q" podUID="ce70492e-c727-4dc1-b2ee-bf278d29a45b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:48:41.717326 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:41.717285 6960 patch_prober.go:29] interesting pod/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-drlj6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:48:41.717572 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:41.717544 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-drlj6" podUID="1ae98fd5-ff16-44e3-80aa-139438c86c78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:48:41.820196 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:41.817429 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-aa6736/distinct-w1-aa6736-kserve-b86d7667b-smcbz" podUID="58c23db7-11cb-4ce2-a67e-fe698e89c71d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:48:41.820196 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:41.818588 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-aa6736/simulator-w1-aa6736-kserve-76985dcd5b-njvnn" podUID="943f7840-8d54-4845-b7ec-c0da72ba9c67" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 30 18:48:41.838918 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:41.838885 6960 patch_prober.go:29] interesting pod/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-frrjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 18:48:41.839492 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:41.839463 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-frrjm" podUID="a5cb9eed-a657-4907-9765-9114f12fb473" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 18:48:41.839712 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:41.839026 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-2-w3-7a8fa2-kserve-677bb44d8d-fg55k" podUID="e8701242-b59c-43b1-9c0a-011635edd419" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 30 18:48:41.911322 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:41.911266 6960 patch_prober.go:29] interesting pod/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-vjb5b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:48:41.936171 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:41.911615 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-vjb5b" podUID="160390da-b35c-4030-9fff-d93afbcf9bad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:48:42.038337 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:42.022434 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-2-w2-8a92cc-kserve-6cc4b9c974-n7cd2" podUID="e5fd8f33-8662-4f90-9817-e601f694bf1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:48:42.229761 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:42.221206 6960 patch_prober.go:29] interesting pod/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-hj7n6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:48:42.229761 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:42.221271 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-hj7n6" podUID="71c3b467-e87a-40eb-ad20-7175b0db9e49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:48:42.541228 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:42.513795 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7a8fa2/simulator-w3-7a8fa2-kserve-78587cb8fc-6pvsz" podUID="cc64ebb9-44e9-435a-9db0-d53ce5f139ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:48:42.541228 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:42.524451 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7a8fa2/premium-w3-7a8fa2-kserve-6b6c5f6fb8-n5wtq" podUID="f51fd746-cd85-4608-96a6-f54afa0ec34d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:48:42.597187 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:42.588372 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/simulator-w2-8a92cc-kserve-fb77d84bc-nqf9q" podUID="ce70492e-c727-4dc1-b2ee-bf278d29a45b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:48:42.669167 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:42.651374 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-aa6736/premium-w1-aa6736-kserve-6f986b66f7-nkfrg" podUID="84ca40c5-7033-48c4-bcec-e9ac7c034a4e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:48:42.712737 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:42.712701 6960 patch_prober.go:29] interesting pod/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-frrjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 18:48:42.713014 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:42.712983 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-frrjm" podUID="a5cb9eed-a657-4907-9765-9114f12fb473" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 18:48:42.867172 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:42.860076 6960 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-2-w2-8a92cc-kserve-6cc4b9c974-n7cd2" podUID="e5fd8f33-8662-4f90-9817-e601f694bf1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:48:42.907548 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:42.907502 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-aa6736/distinct-w1-aa6736-kserve-b86d7667b-smcbz" podUID="58c23db7-11cb-4ce2-a67e-fe698e89c71d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 18:48:42.920169 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:42.908105 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-aa6736/simulator-w1-aa6736-kserve-76985dcd5b-njvnn" podUID="943f7840-8d54-4845-b7ec-c0da72ba9c67" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 30 18:48:42.951114 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:42.949246 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-2-w3-7a8fa2-kserve-677bb44d8d-fg55k" podUID="e8701242-b59c-43b1-9c0a-011635edd419" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 30 18:48:43.040170 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:43.038908 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-2-w2-8a92cc-kserve-6cc4b9c974-n7cd2" podUID="e5fd8f33-8662-4f90-9817-e601f694bf1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:48:43.052457 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:43.049759 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-2-w2-8a92cc-kserve-6cc4b9c974-n7cd2" Oct 30 18:48:43.642203 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:43.634455 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7a8fa2/premium-w3-7a8fa2-kserve-6b6c5f6fb8-n5wtq" podUID="f51fd746-cd85-4608-96a6-f54afa0ec34d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:48:43.659233 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:43.654401 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7a8fa2/simulator-w3-7a8fa2-kserve-78587cb8fc-6pvsz" podUID="cc64ebb9-44e9-435a-9db0-d53ce5f139ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 30 18:48:43.820171 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:43.816415 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/simulator-w2-8a92cc-kserve-fb77d84bc-nqf9q" podUID="ce70492e-c727-4dc1-b2ee-bf278d29a45b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 18:48:43.854211 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:43.816915 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-aa6736/premium-w1-aa6736-kserve-6f986b66f7-nkfrg" podUID="84ca40c5-7033-48c4-bcec-e9ac7c034a4e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 30 18:48:43.970229 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:43.920206 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-aa6736/distinct-w1-aa6736-kserve-b86d7667b-smcbz" podUID="58c23db7-11cb-4ce2-a67e-fe698e89c71d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:48:43.970229 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:43.945318 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-aa6736/simulator-w1-aa6736-kserve-76985dcd5b-njvnn" podUID="943f7840-8d54-4845-b7ec-c0da72ba9c67" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:48:44.050691 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:44.050639 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-2-w3-7a8fa2-kserve-677bb44d8d-fg55k" podUID="e8701242-b59c-43b1-9c0a-011635edd419" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 30 18:48:44.084599 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:44.054392 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:44.084599 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:44.055827 6960 patch_prober.go:29] interesting pod/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-frrjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 18:48:44.084599 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:44.055872 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-frrjm" podUID="a5cb9eed-a657-4907-9765-9114f12fb473" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 18:48:44.084599 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:44.057082 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-2-w2-8a92cc-kserve-6cc4b9c974-n7cd2" podUID="e5fd8f33-8662-4f90-9817-e601f694bf1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 18:48:44.431668 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:44.431641 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-aa6736/premium-w1-aa6736-kserve-6f986b66f7-nkfrg" Oct 30 18:48:44.553495 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:44.553466 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-2-w3-7a8fa2-kserve-677bb44d8d-fg55k" Oct 30 18:48:44.558618 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:44.558595 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7a8fa2/simulator-w3-7a8fa2-kserve-78587cb8fc-6pvsz" Oct 30 18:48:44.656168 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:44.655819 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7a8fa2/premium-w3-7a8fa2-kserve-6b6c5f6fb8-n5wtq" podUID="f51fd746-cd85-4608-96a6-f54afa0ec34d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:48:44.791712 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:44.791602 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:44.800693 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:44.799483 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-2-w2-8a92cc-kserve-6cc4b9c974-n7cd2" Oct 30 18:48:44.800993 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:44.800965 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8a92cc/simulator-w2-8a92cc-kserve-fb77d84bc-nqf9q" Oct 30 18:48:44.801174 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:44.801162 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-aa6736/distinct-w1-aa6736-kserve-b86d7667b-smcbz" Oct 30 18:48:44.801339 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:44.801329 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-aa6736/simulator-w1-aa6736-kserve-76985dcd5b-njvnn" Oct 30 18:48:44.809393 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:44.809367 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-7a8fa2/premium-w3-7a8fa2-kserve-6b6c5f6fb8-n5wtq" Oct 30 18:48:45.089171 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:45.089060 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:45.420468 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:45.420387 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-frrjm" Oct 30 18:48:45.420800 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:45.420782 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-frrjm" Oct 30 18:48:45.421870 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:45.421851 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-frrjm" Oct 30 18:48:45.691236 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:45.690273 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:45.854880 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:45.854835 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-58158b-b9798649f-g6c5j"] Oct 30 18:48:45.860400 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:45.860377 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-58158b-b9798649f-g6c5j" Oct 30 18:48:45.866869 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:45.866829 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-58158b-b9798649f-g6c5j"] Oct 30 18:48:45.871177 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:45.871122 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-58158b\"" Oct 30 18:48:45.919860 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:45.919830 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:45.999432 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:45.992131 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6900142-912c-4ac1-a0d6-cb7e97e0fad5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-58158b-b9798649f-g6c5j\" (UID: \"c6900142-912c-4ac1-a0d6-cb7e97e0fad5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-58158b-b9798649f-g6c5j" Oct 30 18:48:45.999432 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:45.992297 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cjhg\" (UniqueName: \"kubernetes.io/projected/c6900142-912c-4ac1-a0d6-cb7e97e0fad5-kube-api-access-7cjhg\") pod \"payload-pre-processing-e2e-api-a-58158b-b9798649f-g6c5j\" (UID: \"c6900142-912c-4ac1-a0d6-cb7e97e0fad5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-58158b-b9798649f-g6c5j" Oct 30 18:48:46.108431 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:46.100664 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6900142-912c-4ac1-a0d6-cb7e97e0fad5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-58158b-b9798649f-g6c5j\" (UID: \"c6900142-912c-4ac1-a0d6-cb7e97e0fad5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-58158b-b9798649f-g6c5j" Oct 30 18:48:46.130383 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:46.109488 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cjhg\" (UniqueName: \"kubernetes.io/projected/c6900142-912c-4ac1-a0d6-cb7e97e0fad5-kube-api-access-7cjhg\") pod \"payload-pre-processing-e2e-api-a-58158b-b9798649f-g6c5j\" (UID: \"c6900142-912c-4ac1-a0d6-cb7e97e0fad5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-58158b-b9798649f-g6c5j" Oct 30 18:48:46.130383 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:46.129094 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6900142-912c-4ac1-a0d6-cb7e97e0fad5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-58158b-b9798649f-g6c5j\" (UID: \"c6900142-912c-4ac1-a0d6-cb7e97e0fad5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-58158b-b9798649f-g6c5j" Oct 30 18:48:46.183104 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:46.183031 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cjhg\" (UniqueName: \"kubernetes.io/projected/c6900142-912c-4ac1-a0d6-cb7e97e0fad5-kube-api-access-7cjhg\") pod \"payload-pre-processing-e2e-api-a-58158b-b9798649f-g6c5j\" (UID: \"c6900142-912c-4ac1-a0d6-cb7e97e0fad5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-58158b-b9798649f-g6c5j" Oct 30 18:48:46.479758 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:46.479445 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-58158b-b9798649f-g6c5j" Oct 30 18:48:46.849833 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:46.849794 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:46.883438 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:46.883401 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-58158b-b9798649f-g6c5j"] Oct 30 18:48:46.889769 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:48:46.889739 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6900142_912c_4ac1_a0d6_cb7e97e0fad5.slice/crio-869b6bff71d7ea1172c35305e859f006335a66fe6ad57ff13b476c66c6eb4c5c WatchSource:0}: Error finding container 869b6bff71d7ea1172c35305e859f006335a66fe6ad57ff13b476c66c6eb4c5c: Status 404 returned error can't find the container with id 869b6bff71d7ea1172c35305e859f006335a66fe6ad57ff13b476c66c6eb4c5c Oct 30 18:48:47.494769 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:47.494736 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-4tbpd"] Oct 30 18:48:47.499851 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:47.499814 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-58158b-b9798649f-g6c5j" event={"ID":"c6900142-912c-4ac1-a0d6-cb7e97e0fad5","Type":"ContainerStarted","Data":"e0fa8f04114fcbf5a208101829bd52f26472042a5e8c0a066eeb6c1c57b80fdd"} Oct 30 18:48:47.499851 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:47.499851 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-58158b-b9798649f-g6c5j" Oct 30 18:48:47.500079 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:47.499873 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-58158b-b9798649f-g6c5j" event={"ID":"c6900142-912c-4ac1-a0d6-cb7e97e0fad5","Type":"ContainerStarted","Data":"869b6bff71d7ea1172c35305e859f006335a66fe6ad57ff13b476c66c6eb4c5c"} Oct 30 18:48:47.500079 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:47.499946 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-4tbpd" Oct 30 18:48:47.545828 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:47.545768 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-58158b-b9798649f-g6c5j" podStartSLOduration=2.5457486400000002 podStartE2EDuration="2.54574864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:48:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:48:47.54088425 +0000 UTC m=+895.069915417" watchObservedRunningTime="2026-10-30 18:48:47.54574864 +0000 UTC m=+895.074779783" Oct 30 18:48:47.551238 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:47.551211 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-4tbpd"] Oct 30 18:48:47.653611 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:47.653577 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb97b10a-d1c7-40a0-b182-2377facd98bc-workload-certs\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-4tbpd\" (UID: \"eb97b10a-d1c7-40a0-b182-2377facd98bc\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-4tbpd" Oct 30 18:48:47.653814 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:47.653619 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb97b10a-d1c7-40a0-b182-2377facd98bc-istio-data\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-4tbpd\" (UID: \"eb97b10a-d1c7-40a0-b182-2377facd98bc\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-4tbpd" Oct 30 18:48:47.653814 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:47.653730 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb97b10a-d1c7-40a0-b182-2377facd98bc-istio-token\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-4tbpd\" (UID: \"eb97b10a-d1c7-40a0-b182-2377facd98bc\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-4tbpd" Oct 30 18:48:47.653945 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:47.653921 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb97b10a-d1c7-40a0-b182-2377facd98bc-credential-socket\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-4tbpd\" (UID: \"eb97b10a-d1c7-40a0-b182-2377facd98bc\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-4tbpd" Oct 30 18:48:47.654338 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:47.654284 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb97b10a-d1c7-40a0-b182-2377facd98bc-istio-podinfo\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-4tbpd\" (UID: \"eb97b10a-d1c7-40a0-b182-2377facd98bc\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-4tbpd" Oct 30 18:48:47.654606 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:47.654578 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb97b10a-d1c7-40a0-b182-2377facd98bc-workload-socket\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-4tbpd\" (UID: \"eb97b10a-d1c7-40a0-b182-2377facd98bc\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-4tbpd" Oct 30 18:48:47.654687 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:47.654642 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4gm7\" (UniqueName: \"kubernetes.io/projected/eb97b10a-d1c7-40a0-b182-2377facd98bc-kube-api-access-s4gm7\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-4tbpd\" (UID: \"eb97b10a-d1c7-40a0-b182-2377facd98bc\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-4tbpd" Oct 30 18:48:47.654742 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:47.654720 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb97b10a-d1c7-40a0-b182-2377facd98bc-istio-envoy\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-4tbpd\" (UID: \"eb97b10a-d1c7-40a0-b182-2377facd98bc\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-4tbpd" Oct 30 18:48:47.654798 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:47.654785 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb97b10a-d1c7-40a0-b182-2377facd98bc-istiod-ca-cert\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-4tbpd\" (UID: \"eb97b10a-d1c7-40a0-b182-2377facd98bc\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-4tbpd" Oct 30 18:48:47.684927 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:47.684807 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:47.755917 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:47.755830 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb97b10a-d1c7-40a0-b182-2377facd98bc-istio-envoy\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-4tbpd\" (UID: \"eb97b10a-d1c7-40a0-b182-2377facd98bc\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-4tbpd" Oct 30 18:48:47.755917 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:47.755877 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb97b10a-d1c7-40a0-b182-2377facd98bc-istiod-ca-cert\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-4tbpd\" (UID: \"eb97b10a-d1c7-40a0-b182-2377facd98bc\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-4tbpd" Oct 30 18:48:47.756110 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:47.755948 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb97b10a-d1c7-40a0-b182-2377facd98bc-workload-certs\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-4tbpd\" (UID: \"eb97b10a-d1c7-40a0-b182-2377facd98bc\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-4tbpd" Oct 30 18:48:47.756110 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:47.755963 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb97b10a-d1c7-40a0-b182-2377facd98bc-istio-data\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-4tbpd\" (UID: \"eb97b10a-d1c7-40a0-b182-2377facd98bc\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-4tbpd" Oct 30 18:48:47.756110 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:47.756031 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb97b10a-d1c7-40a0-b182-2377facd98bc-istio-token\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-4tbpd\" (UID: \"eb97b10a-d1c7-40a0-b182-2377facd98bc\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-4tbpd" Oct 30 18:48:47.756341 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:47.756121 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb97b10a-d1c7-40a0-b182-2377facd98bc-credential-socket\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-4tbpd\" (UID: \"eb97b10a-d1c7-40a0-b182-2377facd98bc\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-4tbpd" Oct 30 18:48:47.756341 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:47.756230 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb97b10a-d1c7-40a0-b182-2377facd98bc-istio-podinfo\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-4tbpd\" (UID: \"eb97b10a-d1c7-40a0-b182-2377facd98bc\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-4tbpd" Oct 30 18:48:47.756341 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:47.756307 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb97b10a-d1c7-40a0-b182-2377facd98bc-workload-socket\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-4tbpd\" (UID: \"eb97b10a-d1c7-40a0-b182-2377facd98bc\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-4tbpd" Oct 30 18:48:47.756482 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:47.756350 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4gm7\" (UniqueName: \"kubernetes.io/projected/eb97b10a-d1c7-40a0-b182-2377facd98bc-kube-api-access-s4gm7\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-4tbpd\" (UID: \"eb97b10a-d1c7-40a0-b182-2377facd98bc\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-4tbpd" Oct 30 18:48:47.756542 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:47.756478 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb97b10a-d1c7-40a0-b182-2377facd98bc-istio-data\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-4tbpd\" (UID: \"eb97b10a-d1c7-40a0-b182-2377facd98bc\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-4tbpd" Oct 30 18:48:47.756663 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:47.756639 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb97b10a-d1c7-40a0-b182-2377facd98bc-credential-socket\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-4tbpd\" (UID: \"eb97b10a-d1c7-40a0-b182-2377facd98bc\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-4tbpd" Oct 30 18:48:47.756777 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:47.756754 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb97b10a-d1c7-40a0-b182-2377facd98bc-istiod-ca-cert\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-4tbpd\" (UID: \"eb97b10a-d1c7-40a0-b182-2377facd98bc\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-4tbpd" Oct 30 18:48:47.756841 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:47.756823 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb97b10a-d1c7-40a0-b182-2377facd98bc-workload-socket\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-4tbpd\" (UID: \"eb97b10a-d1c7-40a0-b182-2377facd98bc\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-4tbpd" Oct 30 18:48:47.756902 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:47.756882 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb97b10a-d1c7-40a0-b182-2377facd98bc-workload-certs\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-4tbpd\" (UID: \"eb97b10a-d1c7-40a0-b182-2377facd98bc\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-4tbpd" Oct 30 18:48:47.758612 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:47.758590 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb97b10a-d1c7-40a0-b182-2377facd98bc-istio-envoy\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-4tbpd\" (UID: \"eb97b10a-d1c7-40a0-b182-2377facd98bc\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-4tbpd" Oct 30 18:48:47.760321 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:47.760294 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb97b10a-d1c7-40a0-b182-2377facd98bc-istio-podinfo\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-4tbpd\" (UID: \"eb97b10a-d1c7-40a0-b182-2377facd98bc\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-4tbpd" Oct 30 18:48:47.786275 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:47.786241 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4gm7\" (UniqueName: \"kubernetes.io/projected/eb97b10a-d1c7-40a0-b182-2377facd98bc-kube-api-access-s4gm7\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-4tbpd\" (UID: \"eb97b10a-d1c7-40a0-b182-2377facd98bc\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-4tbpd" Oct 30 18:48:47.788787 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:47.788760 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb97b10a-d1c7-40a0-b182-2377facd98bc-istio-token\") pod \"e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-4tbpd\" (UID: \"eb97b10a-d1c7-40a0-b182-2377facd98bc\") " pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-4tbpd" Oct 30 18:48:47.816699 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:47.816674 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-4tbpd" Oct 30 18:48:48.119183 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:48.119136 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-4tbpd"] Oct 30 18:48:48.119682 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:48:48.119656 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb97b10a_d1c7_40a0_b182_2377facd98bc.slice/crio-07c36d8f9893b9e15495ba10708c4009c3671ba0fc291b87dacac562d32dbb70 WatchSource:0}: Error finding container 07c36d8f9893b9e15495ba10708c4009c3671ba0fc291b87dacac562d32dbb70: Status 404 returned error can't find the container with id 07c36d8f9893b9e15495ba10708c4009c3671ba0fc291b87dacac562d32dbb70 Oct 30 18:48:48.122359 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:48.122331 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:48:48.122419 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:48.122391 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:48:48.122458 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:48.122420 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:48:48.507200 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:48.507062 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-4tbpd" event={"ID":"eb97b10a-d1c7-40a0-b182-2377facd98bc","Type":"ContainerStarted","Data":"fef5f4e4f15f14b3853fd2b5cf147978f49d35303e16d17f941adeb398a3e52f"} Oct 30 18:48:48.507200 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:48.507126 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-4tbpd" event={"ID":"eb97b10a-d1c7-40a0-b182-2377facd98bc","Type":"ContainerStarted","Data":"07c36d8f9893b9e15495ba10708c4009c3671ba0fc291b87dacac562d32dbb70"} Oct 30 18:48:48.817695 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:48.817668 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-4tbpd" Oct 30 18:48:48.839525 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:48.839490 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:49.109911 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:49.109812 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:49.819339 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:49.819310 6960 patch_prober.go:29] interesting pod/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-4tbpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 18:48:49.819731 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:49.819360 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-4tbpd" podUID="eb97b10a-d1c7-40a0-b182-2377facd98bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 18:48:50.220349 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:50.220223 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:50.474228 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:50.469795 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:50.685199 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:50.684534 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:50.817993 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:50.817963 6960 patch_prober.go:29] interesting pod/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-4tbpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 18:48:50.818191 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:50.818014 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-4tbpd" podUID="eb97b10a-d1c7-40a0-b182-2377facd98bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 18:48:50.888682 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:50.888648 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:51.817375 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:51.817342 6960 patch_prober.go:29] interesting pod/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-4tbpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 18:48:51.817542 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:51.817403 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-4tbpd" podUID="eb97b10a-d1c7-40a0-b182-2377facd98bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 18:48:52.358180 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:52.354776 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-4tbpd" podStartSLOduration=5.35475629 podStartE2EDuration="5.35475629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:48:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:48:48.576502963 +0000 UTC m=+896.105534104" watchObservedRunningTime="2026-10-30 18:48:52.35475629 +0000 UTC m=+899.883787429" Oct 30 18:48:52.383658 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:52.383624 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f4b265-66b947c6d4-tqnnc"] Oct 30 18:48:52.388915 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:52.388888 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f4b265-66b947c6d4-tqnnc" Oct 30 18:48:52.405764 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:52.404306 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-f4b265-dockercfg-5cwvg\"" Oct 30 18:48:52.405764 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:52.405005 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:52.408308 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:52.408281 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f4b265-66b947c6d4-tqnnc"] Oct 30 18:48:52.435551 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:52.435521 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzp9r\" (UniqueName: \"kubernetes.io/projected/e66b2d2a-c6e7-4300-b37a-d8e82408fa35-kube-api-access-kzp9r\") pod \"payload-processing-e2e-api-a-f4b265-66b947c6d4-tqnnc\" (UID: \"e66b2d2a-c6e7-4300-b37a-d8e82408fa35\") " pod="openshift-ingress/payload-processing-e2e-api-a-f4b265-66b947c6d4-tqnnc" Oct 30 18:48:52.435756 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:52.435601 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e66b2d2a-c6e7-4300-b37a-d8e82408fa35-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f4b265-66b947c6d4-tqnnc\" (UID: \"e66b2d2a-c6e7-4300-b37a-d8e82408fa35\") " pod="openshift-ingress/payload-processing-e2e-api-a-f4b265-66b947c6d4-tqnnc" Oct 30 18:48:52.453915 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:52.453897 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-f4b265\"" Oct 30 18:48:52.589173 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:52.584796 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzp9r\" (UniqueName: \"kubernetes.io/projected/e66b2d2a-c6e7-4300-b37a-d8e82408fa35-kube-api-access-kzp9r\") pod \"payload-processing-e2e-api-a-f4b265-66b947c6d4-tqnnc\" (UID: \"e66b2d2a-c6e7-4300-b37a-d8e82408fa35\") " pod="openshift-ingress/payload-processing-e2e-api-a-f4b265-66b947c6d4-tqnnc" Oct 30 18:48:52.589173 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:52.584885 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e66b2d2a-c6e7-4300-b37a-d8e82408fa35-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f4b265-66b947c6d4-tqnnc\" (UID: \"e66b2d2a-c6e7-4300-b37a-d8e82408fa35\") " pod="openshift-ingress/payload-processing-e2e-api-a-f4b265-66b947c6d4-tqnnc" Oct 30 18:48:52.606743 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:52.606715 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e66b2d2a-c6e7-4300-b37a-d8e82408fa35-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f4b265-66b947c6d4-tqnnc\" (UID: \"e66b2d2a-c6e7-4300-b37a-d8e82408fa35\") " pod="openshift-ingress/payload-processing-e2e-api-a-f4b265-66b947c6d4-tqnnc" Oct 30 18:48:52.695333 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:52.695230 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzp9r\" (UniqueName: \"kubernetes.io/projected/e66b2d2a-c6e7-4300-b37a-d8e82408fa35-kube-api-access-kzp9r\") pod \"payload-processing-e2e-api-a-f4b265-66b947c6d4-tqnnc\" (UID: \"e66b2d2a-c6e7-4300-b37a-d8e82408fa35\") " pod="openshift-ingress/payload-processing-e2e-api-a-f4b265-66b947c6d4-tqnnc" Oct 30 18:48:52.700907 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:52.700877 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:52.725942 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:52.725915 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-f4b265-dockercfg-5cwvg\"" Oct 30 18:48:52.733800 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:52.733776 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f4b265-66b947c6d4-tqnnc" Oct 30 18:48:52.817679 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:52.817655 6960 patch_prober.go:29] interesting pod/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-4tbpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 18:48:52.817881 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:52.817858 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-4tbpd" podUID="eb97b10a-d1c7-40a0-b182-2377facd98bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 18:48:53.059909 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:53.059874 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f4b265-66b947c6d4-tqnnc"] Oct 30 18:48:53.629984 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:53.629952 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f4b265-66b947c6d4-tqnnc" event={"ID":"e66b2d2a-c6e7-4300-b37a-d8e82408fa35","Type":"ContainerStarted","Data":"8b9d4b0d3935624046afcbc6d4a6fee2cad01bccbd8fb7a1328abb71214bb8a3"} Oct 30 18:48:53.630477 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:53.630000 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f4b265-66b947c6d4-tqnnc" event={"ID":"e66b2d2a-c6e7-4300-b37a-d8e82408fa35","Type":"ContainerStarted","Data":"f84729cf61ff9d2cf5b278168ce6551c899e43f89ce746c905223b6c857b7e56"} Oct 30 18:48:53.630477 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:53.630038 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-f4b265-66b947c6d4-tqnnc" Oct 30 18:48:53.685385 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:53.681659 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-f4b265-66b947c6d4-tqnnc" podStartSLOduration=1.681639057 podStartE2EDuration="1.681639057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:48:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:48:53.667492643 +0000 UTC m=+901.196523785" watchObservedRunningTime="2026-10-30 18:48:53.681639057 +0000 UTC m=+901.210670198" Oct 30 18:48:53.745536 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:53.745507 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:53.818118 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:53.818082 6960 patch_prober.go:29] interesting pod/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-4tbpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 18:48:53.818328 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:53.818179 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-4tbpd" podUID="eb97b10a-d1c7-40a0-b182-2377facd98bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 18:48:53.935936 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:48:53.935857 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 18:48:54.817910 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:54.817867 6960 patch_prober.go:29] interesting pod/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-4tbpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 18:48:54.818348 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:54.817928 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-4tbpd" podUID="eb97b10a-d1c7-40a0-b182-2377facd98bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 18:48:55.031833 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:55.031801 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:55.822525 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:55.822491 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-4tbpd" Oct 30 18:48:55.822931 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:55.822878 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-4tbpd" Oct 30 18:48:55.823779 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:55.823757 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-4tbpd" Oct 30 18:48:56.108053 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:56.107972 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:56.466045 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:56.465959 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:56.747643 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:56.747545 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:56.983808 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:56.983772 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-2aada387/e2e-lifecycle-model-2aada387-kserve-dcfbdc957-48c2n"] Oct 30 18:48:56.989269 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:56.989238 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-2aada387/e2e-lifecycle-model-2aada387-kserve-dcfbdc957-48c2n" Oct 30 18:48:57.011449 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:57.011424 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-2aada387\"/\"openshift-service-ca.crt\"" Oct 30 18:48:57.011569 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:57.011488 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-2aada387\"/\"kube-root-ca.crt\"" Oct 30 18:48:57.018130 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:57.018107 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-2aada387\"/\"e2e-lifecycle-model-2aada387-kserve-self-signed-certs\"" Oct 30 18:48:57.018130 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:57.018124 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-2aada387\"/\"default-dockercfg-89ddm\"" Oct 30 18:48:57.026289 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:57.026265 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-2aada387/e2e-lifecycle-model-2aada387-kserve-dcfbdc957-48c2n"] Oct 30 18:48:57.062840 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:57.062813 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:57.072780 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:57.072751 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c95874c-5d7b-4aeb-b968-06af27377c8e-tmp-dir\") pod \"e2e-lifecycle-model-2aada387-kserve-dcfbdc957-48c2n\" (UID: \"8c95874c-5d7b-4aeb-b968-06af27377c8e\") " pod="ai-tenant-e2e-mt-2aada387/e2e-lifecycle-model-2aada387-kserve-dcfbdc957-48c2n" Oct 30 18:48:57.072930 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:57.072815 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c95874c-5d7b-4aeb-b968-06af27377c8e-model-cache\") pod \"e2e-lifecycle-model-2aada387-kserve-dcfbdc957-48c2n\" (UID: \"8c95874c-5d7b-4aeb-b968-06af27377c8e\") " pod="ai-tenant-e2e-mt-2aada387/e2e-lifecycle-model-2aada387-kserve-dcfbdc957-48c2n" Oct 30 18:48:57.073056 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:57.073036 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c95874c-5d7b-4aeb-b968-06af27377c8e-home\") pod \"e2e-lifecycle-model-2aada387-kserve-dcfbdc957-48c2n\" (UID: \"8c95874c-5d7b-4aeb-b968-06af27377c8e\") " pod="ai-tenant-e2e-mt-2aada387/e2e-lifecycle-model-2aada387-kserve-dcfbdc957-48c2n" Oct 30 18:48:57.073202 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:57.073100 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c95874c-5d7b-4aeb-b968-06af27377c8e-tls-certs\") pod \"e2e-lifecycle-model-2aada387-kserve-dcfbdc957-48c2n\" (UID: \"8c95874c-5d7b-4aeb-b968-06af27377c8e\") " pod="ai-tenant-e2e-mt-2aada387/e2e-lifecycle-model-2aada387-kserve-dcfbdc957-48c2n" Oct 30 18:48:57.073284 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:57.073266 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btdw9\" (UniqueName: \"kubernetes.io/projected/8c95874c-5d7b-4aeb-b968-06af27377c8e-kube-api-access-btdw9\") pod \"e2e-lifecycle-model-2aada387-kserve-dcfbdc957-48c2n\" (UID: \"8c95874c-5d7b-4aeb-b968-06af27377c8e\") " pod="ai-tenant-e2e-mt-2aada387/e2e-lifecycle-model-2aada387-kserve-dcfbdc957-48c2n" Oct 30 18:48:57.073326 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:57.073300 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c95874c-5d7b-4aeb-b968-06af27377c8e-dshm\") pod \"e2e-lifecycle-model-2aada387-kserve-dcfbdc957-48c2n\" (UID: \"8c95874c-5d7b-4aeb-b968-06af27377c8e\") " pod="ai-tenant-e2e-mt-2aada387/e2e-lifecycle-model-2aada387-kserve-dcfbdc957-48c2n" Oct 30 18:48:57.191170 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:57.183115 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c95874c-5d7b-4aeb-b968-06af27377c8e-tmp-dir\") pod \"e2e-lifecycle-model-2aada387-kserve-dcfbdc957-48c2n\" (UID: \"8c95874c-5d7b-4aeb-b968-06af27377c8e\") " pod="ai-tenant-e2e-mt-2aada387/e2e-lifecycle-model-2aada387-kserve-dcfbdc957-48c2n" Oct 30 18:48:57.191170 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:57.183269 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c95874c-5d7b-4aeb-b968-06af27377c8e-model-cache\") pod \"e2e-lifecycle-model-2aada387-kserve-dcfbdc957-48c2n\" (UID: \"8c95874c-5d7b-4aeb-b968-06af27377c8e\") " pod="ai-tenant-e2e-mt-2aada387/e2e-lifecycle-model-2aada387-kserve-dcfbdc957-48c2n" Oct 30 18:48:57.191170 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:57.183476 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c95874c-5d7b-4aeb-b968-06af27377c8e-home\") pod \"e2e-lifecycle-model-2aada387-kserve-dcfbdc957-48c2n\" (UID: \"8c95874c-5d7b-4aeb-b968-06af27377c8e\") " pod="ai-tenant-e2e-mt-2aada387/e2e-lifecycle-model-2aada387-kserve-dcfbdc957-48c2n" Oct 30 18:48:57.191170 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:57.183532 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c95874c-5d7b-4aeb-b968-06af27377c8e-tls-certs\") pod \"e2e-lifecycle-model-2aada387-kserve-dcfbdc957-48c2n\" (UID: \"8c95874c-5d7b-4aeb-b968-06af27377c8e\") " pod="ai-tenant-e2e-mt-2aada387/e2e-lifecycle-model-2aada387-kserve-dcfbdc957-48c2n" Oct 30 18:48:57.191170 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:57.183652 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btdw9\" (UniqueName: \"kubernetes.io/projected/8c95874c-5d7b-4aeb-b968-06af27377c8e-kube-api-access-btdw9\") pod \"e2e-lifecycle-model-2aada387-kserve-dcfbdc957-48c2n\" (UID: \"8c95874c-5d7b-4aeb-b968-06af27377c8e\") " pod="ai-tenant-e2e-mt-2aada387/e2e-lifecycle-model-2aada387-kserve-dcfbdc957-48c2n" Oct 30 18:48:57.191170 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:57.183690 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c95874c-5d7b-4aeb-b968-06af27377c8e-dshm\") pod \"e2e-lifecycle-model-2aada387-kserve-dcfbdc957-48c2n\" (UID: \"8c95874c-5d7b-4aeb-b968-06af27377c8e\") " pod="ai-tenant-e2e-mt-2aada387/e2e-lifecycle-model-2aada387-kserve-dcfbdc957-48c2n" Oct 30 18:48:57.195167 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:57.193555 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c95874c-5d7b-4aeb-b968-06af27377c8e-home\") pod \"e2e-lifecycle-model-2aada387-kserve-dcfbdc957-48c2n\" (UID: \"8c95874c-5d7b-4aeb-b968-06af27377c8e\") " pod="ai-tenant-e2e-mt-2aada387/e2e-lifecycle-model-2aada387-kserve-dcfbdc957-48c2n" Oct 30 18:48:57.195167 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:57.193815 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c95874c-5d7b-4aeb-b968-06af27377c8e-tmp-dir\") pod \"e2e-lifecycle-model-2aada387-kserve-dcfbdc957-48c2n\" (UID: \"8c95874c-5d7b-4aeb-b968-06af27377c8e\") " pod="ai-tenant-e2e-mt-2aada387/e2e-lifecycle-model-2aada387-kserve-dcfbdc957-48c2n" Oct 30 18:48:57.195167 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:57.194051 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c95874c-5d7b-4aeb-b968-06af27377c8e-model-cache\") pod \"e2e-lifecycle-model-2aada387-kserve-dcfbdc957-48c2n\" (UID: \"8c95874c-5d7b-4aeb-b968-06af27377c8e\") " pod="ai-tenant-e2e-mt-2aada387/e2e-lifecycle-model-2aada387-kserve-dcfbdc957-48c2n" Oct 30 18:48:57.204172 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:57.203679 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c95874c-5d7b-4aeb-b968-06af27377c8e-tls-certs\") pod \"e2e-lifecycle-model-2aada387-kserve-dcfbdc957-48c2n\" (UID: \"8c95874c-5d7b-4aeb-b968-06af27377c8e\") " pod="ai-tenant-e2e-mt-2aada387/e2e-lifecycle-model-2aada387-kserve-dcfbdc957-48c2n" Oct 30 18:48:57.231168 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:57.215432 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btdw9\" (UniqueName: \"kubernetes.io/projected/8c95874c-5d7b-4aeb-b968-06af27377c8e-kube-api-access-btdw9\") pod \"e2e-lifecycle-model-2aada387-kserve-dcfbdc957-48c2n\" (UID: \"8c95874c-5d7b-4aeb-b968-06af27377c8e\") " pod="ai-tenant-e2e-mt-2aada387/e2e-lifecycle-model-2aada387-kserve-dcfbdc957-48c2n" Oct 30 18:48:57.245168 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:57.232178 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c95874c-5d7b-4aeb-b968-06af27377c8e-dshm\") pod \"e2e-lifecycle-model-2aada387-kserve-dcfbdc957-48c2n\" (UID: \"8c95874c-5d7b-4aeb-b968-06af27377c8e\") " pod="ai-tenant-e2e-mt-2aada387/e2e-lifecycle-model-2aada387-kserve-dcfbdc957-48c2n" Oct 30 18:48:57.278223 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:57.268065 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:57.305172 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:57.302480 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-2aada387/e2e-lifecycle-model-2aada387-kserve-dcfbdc957-48c2n" Oct 30 18:48:57.499350 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:57.499314 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-58158b-d66855fb7-v6ngx"] Oct 30 18:48:57.504890 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:57.504866 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-58158b-d66855fb7-v6ngx" Oct 30 18:48:57.512501 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:57.512478 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-58158b-serving-cert\"" Oct 30 18:48:57.514644 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:57.514614 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:57.532055 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:57.531992 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-58158b-d66855fb7-v6ngx"] Oct 30 18:48:57.591025 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:57.590972 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrmqh\" (UniqueName: \"kubernetes.io/projected/fb28fce6-3142-4677-9cec-a018e5764832-kube-api-access-nrmqh\") pod \"maas-api-e2e-api-a-58158b-d66855fb7-v6ngx\" (UID: \"fb28fce6-3142-4677-9cec-a018e5764832\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-58158b-d66855fb7-v6ngx" Oct 30 18:48:57.591274 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:57.591206 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb28fce6-3142-4677-9cec-a018e5764832-openshift-service-ca\") pod \"maas-api-e2e-api-a-58158b-d66855fb7-v6ngx\" (UID: \"fb28fce6-3142-4677-9cec-a018e5764832\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-58158b-d66855fb7-v6ngx" Oct 30 18:48:57.591274 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:57.591258 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb28fce6-3142-4677-9cec-a018e5764832-maas-api-tls\") pod \"maas-api-e2e-api-a-58158b-d66855fb7-v6ngx\" (UID: \"fb28fce6-3142-4677-9cec-a018e5764832\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-58158b-d66855fb7-v6ngx" Oct 30 18:48:57.591402 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:57.591344 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb28fce6-3142-4677-9cec-a018e5764832-metrics-certs\") pod \"maas-api-e2e-api-a-58158b-d66855fb7-v6ngx\" (UID: \"fb28fce6-3142-4677-9cec-a018e5764832\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-58158b-d66855fb7-v6ngx" Oct 30 18:48:57.652199 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:48:57.652171 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8c95874c_5d7b_4aeb_b968_06af27377c8e.slice/crio-bf2325986500fd10141f10ae757dd7f7889e6eaab1978b666699c695be1bf6ef WatchSource:0}: Error finding container bf2325986500fd10141f10ae757dd7f7889e6eaab1978b666699c695be1bf6ef: Status 404 returned error can't find the container with id bf2325986500fd10141f10ae757dd7f7889e6eaab1978b666699c695be1bf6ef Oct 30 18:48:57.652437 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:57.652413 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-2aada387/e2e-lifecycle-model-2aada387-kserve-dcfbdc957-48c2n"] Oct 30 18:48:57.692829 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:57.692804 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb28fce6-3142-4677-9cec-a018e5764832-openshift-service-ca\") pod \"maas-api-e2e-api-a-58158b-d66855fb7-v6ngx\" (UID: \"fb28fce6-3142-4677-9cec-a018e5764832\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-58158b-d66855fb7-v6ngx" Oct 30 18:48:57.692934 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:57.692847 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb28fce6-3142-4677-9cec-a018e5764832-maas-api-tls\") pod \"maas-api-e2e-api-a-58158b-d66855fb7-v6ngx\" (UID: \"fb28fce6-3142-4677-9cec-a018e5764832\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-58158b-d66855fb7-v6ngx" Oct 30 18:48:57.692934 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:57.692911 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb28fce6-3142-4677-9cec-a018e5764832-metrics-certs\") pod \"maas-api-e2e-api-a-58158b-d66855fb7-v6ngx\" (UID: \"fb28fce6-3142-4677-9cec-a018e5764832\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-58158b-d66855fb7-v6ngx" Oct 30 18:48:57.693094 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:57.693076 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrmqh\" (UniqueName: \"kubernetes.io/projected/fb28fce6-3142-4677-9cec-a018e5764832-kube-api-access-nrmqh\") pod \"maas-api-e2e-api-a-58158b-d66855fb7-v6ngx\" (UID: \"fb28fce6-3142-4677-9cec-a018e5764832\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-58158b-d66855fb7-v6ngx" Oct 30 18:48:57.693610 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:57.693586 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb28fce6-3142-4677-9cec-a018e5764832-openshift-service-ca\") pod \"maas-api-e2e-api-a-58158b-d66855fb7-v6ngx\" (UID: \"fb28fce6-3142-4677-9cec-a018e5764832\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-58158b-d66855fb7-v6ngx" Oct 30 18:48:57.695843 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:57.695817 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb28fce6-3142-4677-9cec-a018e5764832-metrics-certs\") pod \"maas-api-e2e-api-a-58158b-d66855fb7-v6ngx\" (UID: \"fb28fce6-3142-4677-9cec-a018e5764832\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-58158b-d66855fb7-v6ngx" Oct 30 18:48:57.695924 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:57.695892 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb28fce6-3142-4677-9cec-a018e5764832-maas-api-tls\") pod \"maas-api-e2e-api-a-58158b-d66855fb7-v6ngx\" (UID: \"fb28fce6-3142-4677-9cec-a018e5764832\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-58158b-d66855fb7-v6ngx" Oct 30 18:48:57.715183 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:57.715124 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:57.719904 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:57.719879 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrmqh\" (UniqueName: \"kubernetes.io/projected/fb28fce6-3142-4677-9cec-a018e5764832-kube-api-access-nrmqh\") pod \"maas-api-e2e-api-a-58158b-d66855fb7-v6ngx\" (UID: \"fb28fce6-3142-4677-9cec-a018e5764832\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-58158b-d66855fb7-v6ngx" Oct 30 18:48:57.819796 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:57.819767 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-58158b-d66855fb7-v6ngx" Oct 30 18:48:57.998253 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:57.992189 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:58.133539 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:58.132921 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-58158b-d66855fb7-v6ngx"] Oct 30 18:48:58.136316 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:58.136190 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:58.143497 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:48:58.143468 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb28fce6_3142_4677_9cec_a018e5764832.slice/crio-4f4deaafbe8439c2d9ce55cced8870eba6f164d518396e8de26c47d957ad1600 WatchSource:0}: Error finding container 4f4deaafbe8439c2d9ce55cced8870eba6f164d518396e8de26c47d957ad1600: Status 404 returned error can't find the container with id 4f4deaafbe8439c2d9ce55cced8870eba6f164d518396e8de26c47d957ad1600 Oct 30 18:48:58.535225 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:58.531824 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-58158b-b9798649f-g6c5j" Oct 30 18:48:58.673739 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:58.673703 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-58158b-d66855fb7-v6ngx" event={"ID":"fb28fce6-3142-4677-9cec-a018e5764832","Type":"ContainerStarted","Data":"4f4deaafbe8439c2d9ce55cced8870eba6f164d518396e8de26c47d957ad1600"} Oct 30 18:48:58.676969 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:58.676938 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-2aada387/e2e-lifecycle-model-2aada387-kserve-dcfbdc957-48c2n" event={"ID":"8c95874c-5d7b-4aeb-b968-06af27377c8e","Type":"ContainerStarted","Data":"9bb352a012907d22139064c0a28767116c4c0d317a5ca9ba5567d79abb6a6e0b"} Oct 30 18:48:58.677116 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:58.676983 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-2aada387/e2e-lifecycle-model-2aada387-kserve-dcfbdc957-48c2n" event={"ID":"8c95874c-5d7b-4aeb-b968-06af27377c8e","Type":"ContainerStarted","Data":"bf2325986500fd10141f10ae757dd7f7889e6eaab1978b666699c695be1bf6ef"} Oct 30 18:48:58.718971 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:58.718899 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-2aada387/e2e-lifecycle-model-2aada387-kserve-dcfbdc957-48c2n" podStartSLOduration=2.7188764279999997 podStartE2EDuration="2.718876428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:48:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:48:58.703038968 +0000 UTC m=+906.232070106" watchObservedRunningTime="2026-10-30 18:48:58.718876428 +0000 UTC m=+906.247907567" Oct 30 18:48:58.945248 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:58.945210 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:59.303874 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:59.303832 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-2aada387/e2e-lifecycle-model-2aada387-kserve-dcfbdc957-48c2n" Oct 30 18:48:59.690734 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:59.690697 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-58158b-d66855fb7-v6ngx" event={"ID":"fb28fce6-3142-4677-9cec-a018e5764832","Type":"ContainerStarted","Data":"10b44180689446f8d8f28aebbf8a004e285e878ff9f24c9e02bd3a791b4e02c0"} Oct 30 18:48:59.690941 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:59.690819 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-58158b-d66855fb7-v6ngx" Oct 30 18:48:59.732823 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:48:59.732763 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-58158b-d66855fb7-v6ngx" podStartSLOduration=2.159905389 podStartE2EDuration="2.732744563s" totalImagesPullingTime="572.839174ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:48:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:48:59.731084079 +0000 UTC m=+907.260115254" watchObservedRunningTime="2026-10-30 18:48:59.732744563 +0000 UTC m=+907.261775711" Oct 30 18:49:00.173883 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:00.173611 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:01.609557 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:01.609524 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:01.971453 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:01.971373 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:02.120320 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:02.120278 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:02.729631 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:02.729577 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:02.872574 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:02.872538 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:03.202597 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:03.202559 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:03.459377 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:03.447128 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:04.076544 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:04.076511 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:04.676319 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:04.676292 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-f4b265-66b947c6d4-tqnnc" Oct 30 18:49:05.416649 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:05.416611 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:06.614745 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:06.614711 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:06.711259 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:06.711235 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-58158b-d66855fb7-v6ngx" Oct 30 18:49:07.303411 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:07.303377 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-2aada387/e2e-lifecycle-model-2aada387-kserve-dcfbdc957-48c2n" Oct 30 18:49:07.316268 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:07.316243 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-2aada387/e2e-lifecycle-model-2aada387-kserve-dcfbdc957-48c2n" Oct 30 18:49:07.417979 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:07.417942 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:07.647757 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:07.647719 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:07.820304 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:07.820275 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-2aada387/e2e-lifecycle-model-2aada387-kserve-dcfbdc957-48c2n" Oct 30 18:49:08.162628 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:08.162593 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:08.502894 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:08.502792 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:09.307312 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:09.307273 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:10.286308 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:10.286274 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:11.348369 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:11.348334 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:12.112338 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:12.112306 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f44051-7c85cd7f6f-q4tm9"] Oct 30 18:49:12.116753 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:12.116730 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f44051-7c85cd7f6f-q4tm9" Oct 30 18:49:12.122897 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:12.122874 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-f44051-serving-cert\"" Oct 30 18:49:12.129737 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:12.129709 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f44051-7c85cd7f6f-q4tm9"] Oct 30 18:49:12.201163 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:12.201118 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:12.213476 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:12.213445 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09ddad39-4b6f-47ef-9c29-1353981233dc-openshift-service-ca\") pod \"maas-api-e2e-api-b-f44051-7c85cd7f6f-q4tm9\" (UID: \"09ddad39-4b6f-47ef-9c29-1353981233dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f44051-7c85cd7f6f-q4tm9" Oct 30 18:49:12.213617 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:12.213542 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09ddad39-4b6f-47ef-9c29-1353981233dc-maas-api-tls\") pod \"maas-api-e2e-api-b-f44051-7c85cd7f6f-q4tm9\" (UID: \"09ddad39-4b6f-47ef-9c29-1353981233dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f44051-7c85cd7f6f-q4tm9" Oct 30 18:49:12.213617 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:12.213604 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09ddad39-4b6f-47ef-9c29-1353981233dc-metrics-certs\") pod \"maas-api-e2e-api-b-f44051-7c85cd7f6f-q4tm9\" (UID: \"09ddad39-4b6f-47ef-9c29-1353981233dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f44051-7c85cd7f6f-q4tm9" Oct 30 18:49:12.213787 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:12.213774 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slmj8\" (UniqueName: \"kubernetes.io/projected/09ddad39-4b6f-47ef-9c29-1353981233dc-kube-api-access-slmj8\") pod \"maas-api-e2e-api-b-f44051-7c85cd7f6f-q4tm9\" (UID: \"09ddad39-4b6f-47ef-9c29-1353981233dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f44051-7c85cd7f6f-q4tm9" Oct 30 18:49:12.305337 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:12.305294 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:12.314754 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:12.314723 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slmj8\" (UniqueName: \"kubernetes.io/projected/09ddad39-4b6f-47ef-9c29-1353981233dc-kube-api-access-slmj8\") pod \"maas-api-e2e-api-b-f44051-7c85cd7f6f-q4tm9\" (UID: \"09ddad39-4b6f-47ef-9c29-1353981233dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f44051-7c85cd7f6f-q4tm9" Oct 30 18:49:12.314926 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:12.314834 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09ddad39-4b6f-47ef-9c29-1353981233dc-openshift-service-ca\") pod \"maas-api-e2e-api-b-f44051-7c85cd7f6f-q4tm9\" (UID: \"09ddad39-4b6f-47ef-9c29-1353981233dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f44051-7c85cd7f6f-q4tm9" Oct 30 18:49:12.314926 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:12.314867 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09ddad39-4b6f-47ef-9c29-1353981233dc-maas-api-tls\") pod \"maas-api-e2e-api-b-f44051-7c85cd7f6f-q4tm9\" (UID: \"09ddad39-4b6f-47ef-9c29-1353981233dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f44051-7c85cd7f6f-q4tm9" Oct 30 18:49:12.314926 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:12.314914 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09ddad39-4b6f-47ef-9c29-1353981233dc-metrics-certs\") pod \"maas-api-e2e-api-b-f44051-7c85cd7f6f-q4tm9\" (UID: \"09ddad39-4b6f-47ef-9c29-1353981233dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f44051-7c85cd7f6f-q4tm9" Oct 30 18:49:12.315048 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:49:12.314989 6960 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-f44051-serving-cert: secret "maas-api-e2e-api-b-f44051-serving-cert" not found Oct 30 18:49:12.315097 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:49:12.315085 6960 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09ddad39-4b6f-47ef-9c29-1353981233dc-maas-api-tls podName:09ddad39-4b6f-47ef-9c29-1353981233dc nodeName:}" failed. No retries permitted until 2026-10-30 18:49:12.815061459 +0000 UTC m=+920.344092590 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/09ddad39-4b6f-47ef-9c29-1353981233dc-maas-api-tls") pod "maas-api-e2e-api-b-f44051-7c85cd7f6f-q4tm9" (UID: "09ddad39-4b6f-47ef-9c29-1353981233dc") : secret "maas-api-e2e-api-b-f44051-serving-cert" not found Oct 30 18:49:12.315651 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:12.315629 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09ddad39-4b6f-47ef-9c29-1353981233dc-openshift-service-ca\") pod \"maas-api-e2e-api-b-f44051-7c85cd7f6f-q4tm9\" (UID: \"09ddad39-4b6f-47ef-9c29-1353981233dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f44051-7c85cd7f6f-q4tm9" Oct 30 18:49:12.317619 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:12.317602 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09ddad39-4b6f-47ef-9c29-1353981233dc-metrics-certs\") pod \"maas-api-e2e-api-b-f44051-7c85cd7f6f-q4tm9\" (UID: \"09ddad39-4b6f-47ef-9c29-1353981233dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f44051-7c85cd7f6f-q4tm9" Oct 30 18:49:12.329927 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:12.329902 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slmj8\" (UniqueName: \"kubernetes.io/projected/09ddad39-4b6f-47ef-9c29-1353981233dc-kube-api-access-slmj8\") pod \"maas-api-e2e-api-b-f44051-7c85cd7f6f-q4tm9\" (UID: \"09ddad39-4b6f-47ef-9c29-1353981233dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f44051-7c85cd7f6f-q4tm9" Oct 30 18:49:12.589011 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:12.588969 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:12.828221 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:12.828170 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09ddad39-4b6f-47ef-9c29-1353981233dc-maas-api-tls\") pod \"maas-api-e2e-api-b-f44051-7c85cd7f6f-q4tm9\" (UID: \"09ddad39-4b6f-47ef-9c29-1353981233dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f44051-7c85cd7f6f-q4tm9" Oct 30 18:49:12.845302 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:12.845196 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09ddad39-4b6f-47ef-9c29-1353981233dc-maas-api-tls\") pod \"maas-api-e2e-api-b-f44051-7c85cd7f6f-q4tm9\" (UID: \"09ddad39-4b6f-47ef-9c29-1353981233dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f44051-7c85cd7f6f-q4tm9" Oct 30 18:49:13.042993 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:13.042955 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f44051-7c85cd7f6f-q4tm9" Oct 30 18:49:13.165547 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:13.165508 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:13.334113 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:13.334087 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f44051-7c85cd7f6f-q4tm9"] Oct 30 18:49:13.335395 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:49:13.335369 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09ddad39_4b6f_47ef_9c29_1353981233dc.slice/crio-3b00d14ad896c63b9fb5bc2861cbc804c9c5e1fa45abcba2dfc0accb24128350 WatchSource:0}: Error finding container 3b00d14ad896c63b9fb5bc2861cbc804c9c5e1fa45abcba2dfc0accb24128350: Status 404 returned error can't find the container with id 3b00d14ad896c63b9fb5bc2861cbc804c9c5e1fa45abcba2dfc0accb24128350 Oct 30 18:49:13.871287 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:13.871249 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f44051-7c85cd7f6f-q4tm9" event={"ID":"09ddad39-4b6f-47ef-9c29-1353981233dc","Type":"ContainerStarted","Data":"9ee5892a2134b1bb679c68fb9d1618559325f6bb28edafcf812a9aadb465318a"} Oct 30 18:49:13.871671 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:13.871309 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f44051-7c85cd7f6f-q4tm9" event={"ID":"09ddad39-4b6f-47ef-9c29-1353981233dc","Type":"ContainerStarted","Data":"3b00d14ad896c63b9fb5bc2861cbc804c9c5e1fa45abcba2dfc0accb24128350"} Oct 30 18:49:13.871671 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:13.871366 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f44051-7c85cd7f6f-q4tm9" Oct 30 18:49:13.914541 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:13.914060 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f44051-7c85cd7f6f-q4tm9" podStartSLOduration=1.520861582 podStartE2EDuration="1.914044172s" totalImagesPullingTime="393.18259ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:49:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:49:13.913106917 +0000 UTC m=+921.442138054" watchObservedRunningTime="2026-10-30 18:49:13.914044172 +0000 UTC m=+921.443075310" Oct 30 18:49:14.048667 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:14.048577 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:15.115733 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:15.115701 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:15.252824 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:15.252791 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:15.762898 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:15.762869 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:15.875993 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:15.875957 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:15.996700 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:15.996660 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:16.277518 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:16.277489 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zgl9b"] Oct 30 18:49:16.296576 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:16.296550 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zgl9b" Oct 30 18:49:16.312175 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:16.310486 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-7h4l8\"" Oct 30 18:49:16.312175 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:16.310557 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 18:49:16.312175 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:16.310639 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 18:49:16.314997 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:16.314966 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zgl9b"] Oct 30 18:49:16.322247 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:16.322222 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 18:49:16.377616 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:16.377582 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:16.403166 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:16.391189 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c25987f-3a86-4c2e-ae7c-41080e0a3e24-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zgl9b\" (UID: \"9c25987f-3a86-4c2e-ae7c-41080e0a3e24\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zgl9b" Oct 30 18:49:16.403486 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:16.403456 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9c25987f-3a86-4c2e-ae7c-41080e0a3e24-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zgl9b\" (UID: \"9c25987f-3a86-4c2e-ae7c-41080e0a3e24\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zgl9b" Oct 30 18:49:16.413358 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:16.413327 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c25987f-3a86-4c2e-ae7c-41080e0a3e24-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zgl9b\" (UID: \"9c25987f-3a86-4c2e-ae7c-41080e0a3e24\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zgl9b" Oct 30 18:49:16.413732 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:16.413711 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9c25987f-3a86-4c2e-ae7c-41080e0a3e24-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zgl9b\" (UID: \"9c25987f-3a86-4c2e-ae7c-41080e0a3e24\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zgl9b" Oct 30 18:49:16.413975 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:16.413957 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9c25987f-3a86-4c2e-ae7c-41080e0a3e24-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zgl9b\" (UID: \"9c25987f-3a86-4c2e-ae7c-41080e0a3e24\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zgl9b" Oct 30 18:49:16.414098 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:16.414084 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sbvz\" (UniqueName: \"kubernetes.io/projected/9c25987f-3a86-4c2e-ae7c-41080e0a3e24-kube-api-access-7sbvz\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zgl9b\" (UID: \"9c25987f-3a86-4c2e-ae7c-41080e0a3e24\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zgl9b" Oct 30 18:49:16.414253 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:16.414235 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9c25987f-3a86-4c2e-ae7c-41080e0a3e24-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zgl9b\" (UID: \"9c25987f-3a86-4c2e-ae7c-41080e0a3e24\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zgl9b" Oct 30 18:49:16.520166 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:16.519176 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9c25987f-3a86-4c2e-ae7c-41080e0a3e24-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zgl9b\" (UID: \"9c25987f-3a86-4c2e-ae7c-41080e0a3e24\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zgl9b" Oct 30 18:49:16.520166 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:16.519283 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sbvz\" (UniqueName: \"kubernetes.io/projected/9c25987f-3a86-4c2e-ae7c-41080e0a3e24-kube-api-access-7sbvz\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zgl9b\" (UID: \"9c25987f-3a86-4c2e-ae7c-41080e0a3e24\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zgl9b" Oct 30 18:49:16.520166 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:16.519325 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9c25987f-3a86-4c2e-ae7c-41080e0a3e24-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zgl9b\" (UID: \"9c25987f-3a86-4c2e-ae7c-41080e0a3e24\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zgl9b" Oct 30 18:49:16.520166 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:16.519444 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c25987f-3a86-4c2e-ae7c-41080e0a3e24-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zgl9b\" (UID: \"9c25987f-3a86-4c2e-ae7c-41080e0a3e24\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zgl9b" Oct 30 18:49:16.520166 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:16.519477 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9c25987f-3a86-4c2e-ae7c-41080e0a3e24-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zgl9b\" (UID: \"9c25987f-3a86-4c2e-ae7c-41080e0a3e24\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zgl9b" Oct 30 18:49:16.520166 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:16.519519 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c25987f-3a86-4c2e-ae7c-41080e0a3e24-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zgl9b\" (UID: \"9c25987f-3a86-4c2e-ae7c-41080e0a3e24\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zgl9b" Oct 30 18:49:16.520166 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:16.519671 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9c25987f-3a86-4c2e-ae7c-41080e0a3e24-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zgl9b\" (UID: \"9c25987f-3a86-4c2e-ae7c-41080e0a3e24\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zgl9b" Oct 30 18:49:16.543569 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:16.542925 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9c25987f-3a86-4c2e-ae7c-41080e0a3e24-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zgl9b\" (UID: \"9c25987f-3a86-4c2e-ae7c-41080e0a3e24\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zgl9b" Oct 30 18:49:16.544612 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:16.544372 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9c25987f-3a86-4c2e-ae7c-41080e0a3e24-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zgl9b\" (UID: \"9c25987f-3a86-4c2e-ae7c-41080e0a3e24\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zgl9b" Oct 30 18:49:16.550946 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:16.547591 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c25987f-3a86-4c2e-ae7c-41080e0a3e24-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zgl9b\" (UID: \"9c25987f-3a86-4c2e-ae7c-41080e0a3e24\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zgl9b" Oct 30 18:49:16.550946 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:16.547878 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9c25987f-3a86-4c2e-ae7c-41080e0a3e24-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zgl9b\" (UID: \"9c25987f-3a86-4c2e-ae7c-41080e0a3e24\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zgl9b" Oct 30 18:49:16.550946 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:16.548571 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9c25987f-3a86-4c2e-ae7c-41080e0a3e24-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zgl9b\" (UID: \"9c25987f-3a86-4c2e-ae7c-41080e0a3e24\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zgl9b" Oct 30 18:49:16.561944 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:16.561878 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sbvz\" (UniqueName: \"kubernetes.io/projected/9c25987f-3a86-4c2e-ae7c-41080e0a3e24-kube-api-access-7sbvz\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zgl9b\" (UID: \"9c25987f-3a86-4c2e-ae7c-41080e0a3e24\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zgl9b" Oct 30 18:49:16.605562 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:16.594777 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c25987f-3a86-4c2e-ae7c-41080e0a3e24-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zgl9b\" (UID: \"9c25987f-3a86-4c2e-ae7c-41080e0a3e24\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zgl9b" Oct 30 18:49:16.611109 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:16.611074 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:16.634263 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:16.634234 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zgl9b" Oct 30 18:49:16.834621 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:16.834590 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:16.954975 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:49:16.954931 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c25987f_3a86_4c2e_ae7c_41080e0a3e24.slice/crio-1a480c9c1637ea27124feda0c49f956d7389cc3a63cbf9e78df53c7183700581 WatchSource:0}: Error finding container 1a480c9c1637ea27124feda0c49f956d7389cc3a63cbf9e78df53c7183700581: Status 404 returned error can't find the container with id 1a480c9c1637ea27124feda0c49f956d7389cc3a63cbf9e78df53c7183700581 Oct 30 18:49:16.955508 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:16.955479 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zgl9b"] Oct 30 18:49:17.174588 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:17.174501 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:17.370385 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:17.370307 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:17.635206 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:17.632788 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:17.904738 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:17.904663 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zgl9b" event={"ID":"9c25987f-3a86-4c2e-ae7c-41080e0a3e24","Type":"ContainerStarted","Data":"1a480c9c1637ea27124feda0c49f956d7389cc3a63cbf9e78df53c7183700581"} Oct 30 18:49:18.324943 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:18.324904 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:19.512557 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:19.512510 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:19.887537 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:19.887508 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f44051-7c85cd7f6f-q4tm9" Oct 30 18:49:20.166261 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:20.166174 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:20.264424 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:20.264392 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:49:20.264549 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:20.264458 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:49:20.426647 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:20.426573 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:20.708971 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:20.708877 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:20.934912 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:20.934868 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zgl9b" event={"ID":"9c25987f-3a86-4c2e-ae7c-41080e0a3e24","Type":"ContainerStarted","Data":"ad3386ab527b9f935dbb792f3fb6567951172f5014c0e8b36b7a4b41629998ac"} Oct 30 18:49:20.984784 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:20.984681 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zgl9b" podStartSLOduration=1.6791167 podStartE2EDuration="4.984666932s" totalImagesPullingTime="3.305550232s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:49:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:49:20.981187198 +0000 UTC m=+928.510218337" watchObservedRunningTime="2026-10-30 18:49:20.984666932 +0000 UTC m=+928.513698070" Oct 30 18:49:21.035434 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:21.035399 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:21.212459 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:21.212412 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:21.488007 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:21.487972 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:21.802078 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:21.801986 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:21.987164 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:21.981773 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zgl9b" Oct 30 18:49:21.987555 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:21.987538 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zgl9b" Oct 30 18:49:22.421024 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:22.420985 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:22.896433 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:22.896376 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:23.025844 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:23.025811 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:23.522580 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:23.522553 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:23.937190 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:49:23.937135 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 18:49:24.405275 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:24.405243 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:25.495435 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:25.495402 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:26.452130 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:26.452093 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:27.313129 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:27.313095 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:27.418548 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:27.418514 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:28.341320 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:28.341282 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:29.333184 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:29.333131 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:29.534479 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:29.534442 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:30.631709 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:30.631678 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:31.718743 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:31.718692 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:31.858796 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:31.858759 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:32.930899 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:32.930864 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:33.826605 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:33.826521 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:33.937009 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:33.936956 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:34.487514 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:34.487479 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:35.002324 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:35.002285 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:36.017770 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:36.017737 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:36.918598 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:36.918565 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:37.141743 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:37.141713 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:37.923575 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:37.923544 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:38.835844 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:38.835794 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:39.186585 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:39.186503 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:39.302552 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:39.302517 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:39.735741 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:39.735691 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:40.553912 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:40.553860 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:41.410128 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:41.410087 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:42.196270 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:42.196218 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:42.901371 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:42.901340 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-tm4gr"] Oct 30 18:49:42.911462 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:42.911435 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-tm4gr" Oct 30 18:49:42.920914 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:42.920889 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-f44051-openshift-default-dockercfg-5wcgd\"" Oct 30 18:49:42.925929 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:42.925903 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-tm4gr"] Oct 30 18:49:43.049167 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:43.037175 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:43.067942 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:43.067908 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4065a953-3883-4ca3-89d2-c81fcbb3444a-istio-envoy\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-tm4gr\" (UID: \"4065a953-3883-4ca3-89d2-c81fcbb3444a\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-tm4gr" Oct 30 18:49:43.068302 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:43.068279 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4065a953-3883-4ca3-89d2-c81fcbb3444a-credential-socket\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-tm4gr\" (UID: \"4065a953-3883-4ca3-89d2-c81fcbb3444a\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-tm4gr" Oct 30 18:49:43.068597 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:43.068578 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4065a953-3883-4ca3-89d2-c81fcbb3444a-istio-podinfo\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-tm4gr\" (UID: \"4065a953-3883-4ca3-89d2-c81fcbb3444a\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-tm4gr" Oct 30 18:49:43.068808 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:43.068792 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4065a953-3883-4ca3-89d2-c81fcbb3444a-istiod-ca-cert\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-tm4gr\" (UID: \"4065a953-3883-4ca3-89d2-c81fcbb3444a\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-tm4gr" Oct 30 18:49:43.068980 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:43.068965 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4065a953-3883-4ca3-89d2-c81fcbb3444a-workload-certs\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-tm4gr\" (UID: \"4065a953-3883-4ca3-89d2-c81fcbb3444a\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-tm4gr" Oct 30 18:49:43.069254 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:43.069233 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4065a953-3883-4ca3-89d2-c81fcbb3444a-istio-data\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-tm4gr\" (UID: \"4065a953-3883-4ca3-89d2-c81fcbb3444a\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-tm4gr" Oct 30 18:49:43.069482 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:43.069463 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckdpp\" (UniqueName: \"kubernetes.io/projected/4065a953-3883-4ca3-89d2-c81fcbb3444a-kube-api-access-ckdpp\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-tm4gr\" (UID: \"4065a953-3883-4ca3-89d2-c81fcbb3444a\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-tm4gr" Oct 30 18:49:43.069645 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:43.069628 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4065a953-3883-4ca3-89d2-c81fcbb3444a-istio-token\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-tm4gr\" (UID: \"4065a953-3883-4ca3-89d2-c81fcbb3444a\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-tm4gr" Oct 30 18:49:43.069802 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:43.069787 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4065a953-3883-4ca3-89d2-c81fcbb3444a-workload-socket\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-tm4gr\" (UID: \"4065a953-3883-4ca3-89d2-c81fcbb3444a\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-tm4gr" Oct 30 18:49:43.171547 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:43.171456 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4065a953-3883-4ca3-89d2-c81fcbb3444a-istio-data\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-tm4gr\" (UID: \"4065a953-3883-4ca3-89d2-c81fcbb3444a\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-tm4gr" Oct 30 18:49:43.172560 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:43.172536 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckdpp\" (UniqueName: \"kubernetes.io/projected/4065a953-3883-4ca3-89d2-c81fcbb3444a-kube-api-access-ckdpp\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-tm4gr\" (UID: \"4065a953-3883-4ca3-89d2-c81fcbb3444a\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-tm4gr" Oct 30 18:49:43.173400 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:43.173377 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4065a953-3883-4ca3-89d2-c81fcbb3444a-istio-token\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-tm4gr\" (UID: \"4065a953-3883-4ca3-89d2-c81fcbb3444a\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-tm4gr" Oct 30 18:49:43.173827 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:43.173805 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4065a953-3883-4ca3-89d2-c81fcbb3444a-workload-socket\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-tm4gr\" (UID: \"4065a953-3883-4ca3-89d2-c81fcbb3444a\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-tm4gr" Oct 30 18:49:43.173939 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:43.173920 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4065a953-3883-4ca3-89d2-c81fcbb3444a-istio-envoy\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-tm4gr\" (UID: \"4065a953-3883-4ca3-89d2-c81fcbb3444a\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-tm4gr" Oct 30 18:49:43.174009 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:43.173979 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4065a953-3883-4ca3-89d2-c81fcbb3444a-credential-socket\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-tm4gr\" (UID: \"4065a953-3883-4ca3-89d2-c81fcbb3444a\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-tm4gr" Oct 30 18:49:43.174113 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:43.174094 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4065a953-3883-4ca3-89d2-c81fcbb3444a-istio-podinfo\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-tm4gr\" (UID: \"4065a953-3883-4ca3-89d2-c81fcbb3444a\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-tm4gr" Oct 30 18:49:43.174234 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:43.174199 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4065a953-3883-4ca3-89d2-c81fcbb3444a-istiod-ca-cert\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-tm4gr\" (UID: \"4065a953-3883-4ca3-89d2-c81fcbb3444a\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-tm4gr" Oct 30 18:49:43.174300 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:43.174258 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4065a953-3883-4ca3-89d2-c81fcbb3444a-workload-certs\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-tm4gr\" (UID: \"4065a953-3883-4ca3-89d2-c81fcbb3444a\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-tm4gr" Oct 30 18:49:43.174757 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:43.174733 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4065a953-3883-4ca3-89d2-c81fcbb3444a-workload-certs\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-tm4gr\" (UID: \"4065a953-3883-4ca3-89d2-c81fcbb3444a\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-tm4gr" Oct 30 18:49:43.174946 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:43.174924 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4065a953-3883-4ca3-89d2-c81fcbb3444a-workload-socket\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-tm4gr\" (UID: \"4065a953-3883-4ca3-89d2-c81fcbb3444a\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-tm4gr" Oct 30 18:49:43.175052 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:43.172431 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4065a953-3883-4ca3-89d2-c81fcbb3444a-istio-data\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-tm4gr\" (UID: \"4065a953-3883-4ca3-89d2-c81fcbb3444a\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-tm4gr" Oct 30 18:49:43.175953 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:43.175931 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4065a953-3883-4ca3-89d2-c81fcbb3444a-istiod-ca-cert\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-tm4gr\" (UID: \"4065a953-3883-4ca3-89d2-c81fcbb3444a\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-tm4gr" Oct 30 18:49:43.177558 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:43.177538 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4065a953-3883-4ca3-89d2-c81fcbb3444a-credential-socket\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-tm4gr\" (UID: \"4065a953-3883-4ca3-89d2-c81fcbb3444a\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-tm4gr" Oct 30 18:49:43.179406 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:43.179375 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4065a953-3883-4ca3-89d2-c81fcbb3444a-istio-envoy\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-tm4gr\" (UID: \"4065a953-3883-4ca3-89d2-c81fcbb3444a\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-tm4gr" Oct 30 18:49:43.180786 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:43.180760 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4065a953-3883-4ca3-89d2-c81fcbb3444a-istio-podinfo\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-tm4gr\" (UID: \"4065a953-3883-4ca3-89d2-c81fcbb3444a\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-tm4gr" Oct 30 18:49:43.185901 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:43.185873 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckdpp\" (UniqueName: \"kubernetes.io/projected/4065a953-3883-4ca3-89d2-c81fcbb3444a-kube-api-access-ckdpp\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-tm4gr\" (UID: \"4065a953-3883-4ca3-89d2-c81fcbb3444a\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-tm4gr" Oct 30 18:49:43.186220 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:43.186201 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4065a953-3883-4ca3-89d2-c81fcbb3444a-istio-token\") pod \"e2e-api-b-f44051-openshift-default-f97b64fbc-tm4gr\" (UID: \"4065a953-3883-4ca3-89d2-c81fcbb3444a\") " pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-tm4gr" Oct 30 18:49:43.232636 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:43.232613 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-tm4gr" Oct 30 18:49:43.519090 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:43.519036 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-tm4gr"] Oct 30 18:49:43.519323 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:49:43.519298 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4065a953_3883_4ca3_89d2_c81fcbb3444a.slice/crio-9a4b9d1d22fc5db0d9e182d0ab33d488fd826c83752dd9811567477e768e674c WatchSource:0}: Error finding container 9a4b9d1d22fc5db0d9e182d0ab33d488fd826c83752dd9811567477e768e674c: Status 404 returned error can't find the container with id 9a4b9d1d22fc5db0d9e182d0ab33d488fd826c83752dd9811567477e768e674c Oct 30 18:49:43.522031 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:43.521998 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:49:43.522134 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:43.522089 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:49:43.522301 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:43.522159 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:49:43.927979 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:43.927472 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:44.169133 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:44.168857 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-tm4gr" event={"ID":"4065a953-3883-4ca3-89d2-c81fcbb3444a","Type":"ContainerStarted","Data":"5952e653f141683cb867ef97c98da801787e1214d2df1a35f1b54b50836b0c76"} Oct 30 18:49:44.169133 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:44.168934 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-tm4gr" event={"ID":"4065a953-3883-4ca3-89d2-c81fcbb3444a","Type":"ContainerStarted","Data":"9a4b9d1d22fc5db0d9e182d0ab33d488fd826c83752dd9811567477e768e674c"} Oct 30 18:49:44.211081 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:44.210967 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-tm4gr" podStartSLOduration=2.210945586 podStartE2EDuration="2.210945586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:49:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:49:44.2040613 +0000 UTC m=+951.733092438" watchObservedRunningTime="2026-10-30 18:49:44.210945586 +0000 UTC m=+951.739976724" Oct 30 18:49:44.233773 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:44.232954 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-tm4gr" Oct 30 18:49:44.235049 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:44.235020 6960 patch_prober.go:29] interesting pod/e2e-api-b-f44051-openshift-default-f97b64fbc-tm4gr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 18:49:44.235228 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:44.235072 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-tm4gr" podUID="4065a953-3883-4ca3-89d2-c81fcbb3444a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 18:49:45.139023 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:45.138989 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:45.233248 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:45.233219 6960 patch_prober.go:29] interesting pod/e2e-api-b-f44051-openshift-default-f97b64fbc-tm4gr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 18:49:45.233418 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:45.233270 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-tm4gr" podUID="4065a953-3883-4ca3-89d2-c81fcbb3444a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 18:49:45.389393 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:45.389301 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:46.236401 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:46.236370 6960 patch_prober.go:29] interesting pod/e2e-api-b-f44051-openshift-default-f97b64fbc-tm4gr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 18:49:46.236580 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:46.236419 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-tm4gr" podUID="4065a953-3883-4ca3-89d2-c81fcbb3444a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 18:49:46.374171 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:46.368457 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:47.232959 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:47.232927 6960 patch_prober.go:29] interesting pod/e2e-api-b-f44051-openshift-default-f97b64fbc-tm4gr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 18:49:47.233534 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:47.232982 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-tm4gr" podUID="4065a953-3883-4ca3-89d2-c81fcbb3444a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 18:49:47.394160 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:47.394106 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:48.233215 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:48.233178 6960 patch_prober.go:29] interesting pod/e2e-api-b-f44051-openshift-default-f97b64fbc-tm4gr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 18:49:48.233661 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:48.233239 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-tm4gr" podUID="4065a953-3883-4ca3-89d2-c81fcbb3444a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 18:49:48.280622 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:48.280588 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:49.234042 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:49.233999 6960 patch_prober.go:29] interesting pod/e2e-api-b-f44051-openshift-default-f97b64fbc-tm4gr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 18:49:49.234566 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:49.234062 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-tm4gr" podUID="4065a953-3883-4ca3-89d2-c81fcbb3444a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 18:49:49.416242 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:49.416207 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:50.237987 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:50.237961 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-tm4gr" Oct 30 18:49:50.238438 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:50.238419 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-tm4gr" Oct 30 18:49:50.238949 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:50.238929 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-tm4gr" Oct 30 18:49:50.379036 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:50.379004 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:51.083361 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:51.083289 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:51.195919 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:51.195885 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:51.565832 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:51.565791 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:52.231681 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:52.231645 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:53.037203 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:53.037169 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:53.302179 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:53.302074 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:53.504775 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:53.504727 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:53.621338 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:53.621290 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:53.933820 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:49:53.933726 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 18:49:54.099300 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:54.099270 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:54.625349 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:54.625318 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:55.514028 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:55.513993 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:56.412651 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:56.412616 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:57.304367 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:57.304328 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:58.212641 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:58.212603 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:58.838136 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:58.838102 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-58158b-b9798649f-g6c5j"] Oct 30 18:49:58.936221 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:58.935653 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:59.059710 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:59.059676 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:59.482050 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:49:59.482012 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:00.297974 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:00.297937 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:01.095663 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:01.095624 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:01.278917 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:01.278878 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:01.395379 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:01.395240 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:01.710008 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:01.709918 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:02.013752 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:02.013665 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:02.568556 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:02.568516 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f4b265-66b947c6d4-tqnnc"] Oct 30 18:50:02.664464 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:02.664432 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:02.957036 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:02.956937 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:03.356653 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:03.356623 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:03.838705 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:03.838643 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-58158b-b9798649f-g6c5j" podUID="c6900142-912c-4ac1-a0d6-cb7e97e0fad5" containerName="payload-pre-processing" containerID="cri-o://e0fa8f04114fcbf5a208101829bd52f26472042a5e8c0a066eeb6c1c57b80fdd" gracePeriod=25 Oct 30 18:50:04.107338 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:04.107275 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-58158b-b9798649f-g6c5j" Oct 30 18:50:04.217465 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:04.217437 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6900142-912c-4ac1-a0d6-cb7e97e0fad5-kube-api-access-7cjhg\" (UniqueName: \"kubernetes.io/projected/c6900142-912c-4ac1-a0d6-cb7e97e0fad5-kube-api-access-7cjhg\") pod \"c6900142-912c-4ac1-a0d6-cb7e97e0fad5\" (UID: \"c6900142-912c-4ac1-a0d6-cb7e97e0fad5\") " Oct 30 18:50:04.217465 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:04.217473 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6900142-912c-4ac1-a0d6-cb7e97e0fad5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6900142-912c-4ac1-a0d6-cb7e97e0fad5-plugins-config-volume\") pod \"c6900142-912c-4ac1-a0d6-cb7e97e0fad5\" (UID: \"c6900142-912c-4ac1-a0d6-cb7e97e0fad5\") " Oct 30 18:50:04.217885 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:04.217858 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6900142-912c-4ac1-a0d6-cb7e97e0fad5-plugins-config-volume" pod "c6900142-912c-4ac1-a0d6-cb7e97e0fad5" (UID: "c6900142-912c-4ac1-a0d6-cb7e97e0fad5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:50:04.218075 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:04.218062 6960 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6900142-912c-4ac1-a0d6-cb7e97e0fad5-plugins-config-volume\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:50:04.219799 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:04.219769 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6900142-912c-4ac1-a0d6-cb7e97e0fad5-kube-api-access-7cjhg" pod "c6900142-912c-4ac1-a0d6-cb7e97e0fad5" (UID: "c6900142-912c-4ac1-a0d6-cb7e97e0fad5"). InnerVolumeSpecName "kube-api-access-7cjhg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:50:04.224682 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:04.224651 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:04.320099 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:04.320058 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cjhg\" (UniqueName: \"kubernetes.io/projected/c6900142-912c-4ac1-a0d6-cb7e97e0fad5-kube-api-access-7cjhg\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:50:04.390290 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:04.390204 6960 generic.go:356] "Generic (PLEG): container finished" podID="c6900142-912c-4ac1-a0d6-cb7e97e0fad5" containerID="e0fa8f04114fcbf5a208101829bd52f26472042a5e8c0a066eeb6c1c57b80fdd" exitCode=0 Oct 30 18:50:04.390290 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:04.390257 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-58158b-b9798649f-g6c5j" Oct 30 18:50:04.390478 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:04.390290 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-58158b-b9798649f-g6c5j" event={"ID":"c6900142-912c-4ac1-a0d6-cb7e97e0fad5","Type":"ContainerDied","Data":"e0fa8f04114fcbf5a208101829bd52f26472042a5e8c0a066eeb6c1c57b80fdd"} Oct 30 18:50:04.390478 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:04.390370 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-58158b-b9798649f-g6c5j" event={"ID":"c6900142-912c-4ac1-a0d6-cb7e97e0fad5","Type":"ContainerDied","Data":"869b6bff71d7ea1172c35305e859f006335a66fe6ad57ff13b476c66c6eb4c5c"} Oct 30 18:50:04.390478 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:04.390392 6960 scope.go:122] "RemoveContainer" containerID="e0fa8f04114fcbf5a208101829bd52f26472042a5e8c0a066eeb6c1c57b80fdd" Oct 30 18:50:04.401656 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:04.401639 6960 scope.go:122] "RemoveContainer" containerID="e0fa8f04114fcbf5a208101829bd52f26472042a5e8c0a066eeb6c1c57b80fdd" Oct 30 18:50:04.401950 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:50:04.401932 6960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0fa8f04114fcbf5a208101829bd52f26472042a5e8c0a066eeb6c1c57b80fdd\": container with ID starting with e0fa8f04114fcbf5a208101829bd52f26472042a5e8c0a066eeb6c1c57b80fdd not found: ID does not exist" containerID="e0fa8f04114fcbf5a208101829bd52f26472042a5e8c0a066eeb6c1c57b80fdd" Oct 30 18:50:04.402003 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:04.401960 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0fa8f04114fcbf5a208101829bd52f26472042a5e8c0a066eeb6c1c57b80fdd"} err="failed to get container status \"e0fa8f04114fcbf5a208101829bd52f26472042a5e8c0a066eeb6c1c57b80fdd\": rpc error: code = NotFound desc = could not find container \"e0fa8f04114fcbf5a208101829bd52f26472042a5e8c0a066eeb6c1c57b80fdd\": container with ID starting with e0fa8f04114fcbf5a208101829bd52f26472042a5e8c0a066eeb6c1c57b80fdd not found: ID does not exist" Oct 30 18:50:04.434098 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:04.433974 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-58158b-b9798649f-g6c5j"] Oct 30 18:50:04.437555 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:04.437529 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-58158b-b9798649f-g6c5j"] Oct 30 18:50:04.593893 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:04.593853 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6900142-912c-4ac1-a0d6-cb7e97e0fad5" path="/var/lib/kubelet/pods/c6900142-912c-4ac1-a0d6-cb7e97e0fad5/volumes" Oct 30 18:50:05.121184 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:05.121111 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:05.920874 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:05.920831 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:06.713491 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:06.713447 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:07.537587 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:07.537537 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:07.569880 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:07.569837 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-f4b265-66b947c6d4-tqnnc" podUID="e66b2d2a-c6e7-4300-b37a-d8e82408fa35" containerName="payload-processing" containerID="cri-o://8b9d4b0d3935624046afcbc6d4a6fee2cad01bccbd8fb7a1328abb71214bb8a3" gracePeriod=25 Oct 30 18:50:07.860047 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:07.860022 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f4b265-66b947c6d4-tqnnc" Oct 30 18:50:07.883000 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:07.882967 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e66b2d2a-c6e7-4300-b37a-d8e82408fa35-kube-api-access-kzp9r\" (UniqueName: \"kubernetes.io/projected/e66b2d2a-c6e7-4300-b37a-d8e82408fa35-kube-api-access-kzp9r\") pod \"e66b2d2a-c6e7-4300-b37a-d8e82408fa35\" (UID: \"e66b2d2a-c6e7-4300-b37a-d8e82408fa35\") " Oct 30 18:50:07.883191 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:07.883045 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e66b2d2a-c6e7-4300-b37a-d8e82408fa35-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e66b2d2a-c6e7-4300-b37a-d8e82408fa35-plugins-config-volume\") pod \"e66b2d2a-c6e7-4300-b37a-d8e82408fa35\" (UID: \"e66b2d2a-c6e7-4300-b37a-d8e82408fa35\") " Oct 30 18:50:07.883431 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:07.883411 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e66b2d2a-c6e7-4300-b37a-d8e82408fa35-plugins-config-volume" pod "e66b2d2a-c6e7-4300-b37a-d8e82408fa35" (UID: "e66b2d2a-c6e7-4300-b37a-d8e82408fa35"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:50:07.883654 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:07.883633 6960 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e66b2d2a-c6e7-4300-b37a-d8e82408fa35-plugins-config-volume\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:50:07.885233 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:07.885211 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e66b2d2a-c6e7-4300-b37a-d8e82408fa35-kube-api-access-kzp9r" pod "e66b2d2a-c6e7-4300-b37a-d8e82408fa35" (UID: "e66b2d2a-c6e7-4300-b37a-d8e82408fa35"). InnerVolumeSpecName "kube-api-access-kzp9r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:50:07.984459 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:07.984427 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzp9r\" (UniqueName: \"kubernetes.io/projected/e66b2d2a-c6e7-4300-b37a-d8e82408fa35-kube-api-access-kzp9r\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:50:08.418983 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:08.418944 6960 generic.go:356] "Generic (PLEG): container finished" podID="e66b2d2a-c6e7-4300-b37a-d8e82408fa35" containerID="8b9d4b0d3935624046afcbc6d4a6fee2cad01bccbd8fb7a1328abb71214bb8a3" exitCode=0 Oct 30 18:50:08.419222 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:08.419029 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f4b265-66b947c6d4-tqnnc" Oct 30 18:50:08.419222 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:08.419036 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f4b265-66b947c6d4-tqnnc" event={"ID":"e66b2d2a-c6e7-4300-b37a-d8e82408fa35","Type":"ContainerDied","Data":"8b9d4b0d3935624046afcbc6d4a6fee2cad01bccbd8fb7a1328abb71214bb8a3"} Oct 30 18:50:08.419222 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:08.419135 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f4b265-66b947c6d4-tqnnc" event={"ID":"e66b2d2a-c6e7-4300-b37a-d8e82408fa35","Type":"ContainerDied","Data":"f84729cf61ff9d2cf5b278168ce6551c899e43f89ce746c905223b6c857b7e56"} Oct 30 18:50:08.419222 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:08.419198 6960 scope.go:122] "RemoveContainer" containerID="8b9d4b0d3935624046afcbc6d4a6fee2cad01bccbd8fb7a1328abb71214bb8a3" Oct 30 18:50:08.435386 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:08.435362 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:08.439113 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:08.439091 6960 scope.go:122] "RemoveContainer" containerID="8b9d4b0d3935624046afcbc6d4a6fee2cad01bccbd8fb7a1328abb71214bb8a3" Oct 30 18:50:08.439440 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:50:08.439411 6960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b9d4b0d3935624046afcbc6d4a6fee2cad01bccbd8fb7a1328abb71214bb8a3\": container with ID starting with 8b9d4b0d3935624046afcbc6d4a6fee2cad01bccbd8fb7a1328abb71214bb8a3 not found: ID does not exist" containerID="8b9d4b0d3935624046afcbc6d4a6fee2cad01bccbd8fb7a1328abb71214bb8a3" Oct 30 18:50:08.439571 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:08.439452 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b9d4b0d3935624046afcbc6d4a6fee2cad01bccbd8fb7a1328abb71214bb8a3"} err="failed to get container status \"8b9d4b0d3935624046afcbc6d4a6fee2cad01bccbd8fb7a1328abb71214bb8a3\": rpc error: code = NotFound desc = could not find container \"8b9d4b0d3935624046afcbc6d4a6fee2cad01bccbd8fb7a1328abb71214bb8a3\": container with ID starting with 8b9d4b0d3935624046afcbc6d4a6fee2cad01bccbd8fb7a1328abb71214bb8a3 not found: ID does not exist" Oct 30 18:50:08.456703 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:08.456668 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f4b265-66b947c6d4-tqnnc"] Oct 30 18:50:08.460832 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:08.460799 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f4b265-66b947c6d4-tqnnc"] Oct 30 18:50:08.593622 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:08.593593 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e66b2d2a-c6e7-4300-b37a-d8e82408fa35" path="/var/lib/kubelet/pods/e66b2d2a-c6e7-4300-b37a-d8e82408fa35/volumes" Oct 30 18:50:09.163330 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:09.163300 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:09.970081 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:09.970050 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:10.746597 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:10.746555 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:11.410578 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:11.410529 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:11.674016 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:11.673926 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:11.871882 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:11.871848 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:11.987882 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:11.987809 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:12.422066 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:12.422013 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:12.535501 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:12.535441 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:12.893442 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:12.893401 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:13.026749 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:13.026690 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ace9e50-45s55"] Oct 30 18:50:13.030077 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:13.030049 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ace9e50-45s55"] Oct 30 18:50:13.793540 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:13.793506 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:14.594979 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:14.594949 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c6b8bf7-c56e-4497-a01c-cdaae9975f44" path="/var/lib/kubelet/pods/6c6b8bf7-c56e-4497-a01c-cdaae9975f44/volumes" Oct 30 18:50:14.841394 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:14.841360 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:15.839692 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:15.822099 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:17.023308 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:17.023263 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:17.684782 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:17.684747 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-58158b-d66855fb7-v6ngx"] Oct 30 18:50:17.685015 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:17.684988 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-58158b-d66855fb7-v6ngx" podUID="fb28fce6-3142-4677-9cec-a018e5764832" containerName="maas-api" containerID="cri-o://10b44180689446f8d8f28aebbf8a004e285e878ff9f24c9e02bd3a791b4e02c0" gracePeriod=30 Oct 30 18:50:17.760916 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:17.760664 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:18.007262 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:17.992236 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:18.224350 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:18.224318 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:18.230697 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:18.230673 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-58158b-d66855fb7-v6ngx" Oct 30 18:50:18.312857 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:18.312822 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb28fce6-3142-4677-9cec-a018e5764832-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb28fce6-3142-4677-9cec-a018e5764832-maas-api-tls\") pod \"fb28fce6-3142-4677-9cec-a018e5764832\" (UID: \"fb28fce6-3142-4677-9cec-a018e5764832\") " Oct 30 18:50:18.313086 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:18.313064 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb28fce6-3142-4677-9cec-a018e5764832-kube-api-access-nrmqh\" (UniqueName: \"kubernetes.io/projected/fb28fce6-3142-4677-9cec-a018e5764832-kube-api-access-nrmqh\") pod \"fb28fce6-3142-4677-9cec-a018e5764832\" (UID: \"fb28fce6-3142-4677-9cec-a018e5764832\") " Oct 30 18:50:18.313188 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:18.313113 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb28fce6-3142-4677-9cec-a018e5764832-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb28fce6-3142-4677-9cec-a018e5764832-openshift-service-ca\") pod \"fb28fce6-3142-4677-9cec-a018e5764832\" (UID: \"fb28fce6-3142-4677-9cec-a018e5764832\") " Oct 30 18:50:18.313376 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:18.313352 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb28fce6-3142-4677-9cec-a018e5764832-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb28fce6-3142-4677-9cec-a018e5764832-metrics-certs\") pod \"fb28fce6-3142-4677-9cec-a018e5764832\" (UID: \"fb28fce6-3142-4677-9cec-a018e5764832\") " Oct 30 18:50:18.313727 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:18.313619 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb28fce6-3142-4677-9cec-a018e5764832-openshift-service-ca" pod "fb28fce6-3142-4677-9cec-a018e5764832" (UID: "fb28fce6-3142-4677-9cec-a018e5764832"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:50:18.314081 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:18.314056 6960 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb28fce6-3142-4677-9cec-a018e5764832-openshift-service-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:50:18.315386 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:18.315363 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb28fce6-3142-4677-9cec-a018e5764832-maas-api-tls" pod "fb28fce6-3142-4677-9cec-a018e5764832" (UID: "fb28fce6-3142-4677-9cec-a018e5764832"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:50:18.315886 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:18.315860 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb28fce6-3142-4677-9cec-a018e5764832-kube-api-access-nrmqh" pod "fb28fce6-3142-4677-9cec-a018e5764832" (UID: "fb28fce6-3142-4677-9cec-a018e5764832"). InnerVolumeSpecName "kube-api-access-nrmqh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:50:18.315975 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:18.315946 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb28fce6-3142-4677-9cec-a018e5764832-metrics-certs" pod "fb28fce6-3142-4677-9cec-a018e5764832" (UID: "fb28fce6-3142-4677-9cec-a018e5764832"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:50:18.344956 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:18.344925 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:18.415543 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:18.415512 6960 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb28fce6-3142-4677-9cec-a018e5764832-maas-api-tls\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:50:18.415543 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:18.415544 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrmqh\" (UniqueName: \"kubernetes.io/projected/fb28fce6-3142-4677-9cec-a018e5764832-kube-api-access-nrmqh\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:50:18.415751 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:18.415561 6960 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb28fce6-3142-4677-9cec-a018e5764832-metrics-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:50:18.495482 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:18.495448 6960 generic.go:356] "Generic (PLEG): container finished" podID="fb28fce6-3142-4677-9cec-a018e5764832" containerID="10b44180689446f8d8f28aebbf8a004e285e878ff9f24c9e02bd3a791b4e02c0" exitCode=0 Oct 30 18:50:18.495669 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:18.495510 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-58158b-d66855fb7-v6ngx" Oct 30 18:50:18.495669 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:18.495532 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-58158b-d66855fb7-v6ngx" event={"ID":"fb28fce6-3142-4677-9cec-a018e5764832","Type":"ContainerDied","Data":"10b44180689446f8d8f28aebbf8a004e285e878ff9f24c9e02bd3a791b4e02c0"} Oct 30 18:50:18.495669 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:18.495586 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-58158b-d66855fb7-v6ngx" event={"ID":"fb28fce6-3142-4677-9cec-a018e5764832","Type":"ContainerDied","Data":"4f4deaafbe8439c2d9ce55cced8870eba6f164d518396e8de26c47d957ad1600"} Oct 30 18:50:18.495669 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:18.495615 6960 scope.go:122] "RemoveContainer" containerID="10b44180689446f8d8f28aebbf8a004e285e878ff9f24c9e02bd3a791b4e02c0" Oct 30 18:50:18.507285 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:18.507268 6960 scope.go:122] "RemoveContainer" containerID="10b44180689446f8d8f28aebbf8a004e285e878ff9f24c9e02bd3a791b4e02c0" Oct 30 18:50:18.507567 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:50:18.507549 6960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10b44180689446f8d8f28aebbf8a004e285e878ff9f24c9e02bd3a791b4e02c0\": container with ID starting with 10b44180689446f8d8f28aebbf8a004e285e878ff9f24c9e02bd3a791b4e02c0 not found: ID does not exist" containerID="10b44180689446f8d8f28aebbf8a004e285e878ff9f24c9e02bd3a791b4e02c0" Oct 30 18:50:18.507644 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:18.507574 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10b44180689446f8d8f28aebbf8a004e285e878ff9f24c9e02bd3a791b4e02c0"} err="failed to get container status \"10b44180689446f8d8f28aebbf8a004e285e878ff9f24c9e02bd3a791b4e02c0\": rpc error: code = NotFound desc = could not find container \"10b44180689446f8d8f28aebbf8a004e285e878ff9f24c9e02bd3a791b4e02c0\": container with ID starting with 10b44180689446f8d8f28aebbf8a004e285e878ff9f24c9e02bd3a791b4e02c0 not found: ID does not exist" Oct 30 18:50:18.519941 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:18.519919 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-58158b-d66855fb7-v6ngx"] Oct 30 18:50:18.525263 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:18.525241 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-58158b-d66855fb7-v6ngx"] Oct 30 18:50:18.594491 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:18.594424 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb28fce6-3142-4677-9cec-a018e5764832" path="/var/lib/kubelet/pods/fb28fce6-3142-4677-9cec-a018e5764832/volumes" Oct 30 18:50:18.810131 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:18.810098 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:19.135218 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:19.135182 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:19.257316 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:19.257284 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:19.384860 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:19.384823 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:19.535282 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:19.535203 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:20.504783 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:20.504745 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:21.663409 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:21.663368 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:22.416307 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:22.416268 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:23.217749 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:23.217711 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:23.933278 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:50:23.933237 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 18:50:24.018677 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:24.018646 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:24.892246 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:24.892138 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:25.492835 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:25.492795 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:25.704879 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:25.704836 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:25.914053 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:25.914020 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:26.048168 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:26.039649 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:26.605107 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:26.605075 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2aada387-6db6c946b-g28xh"] Oct 30 18:50:26.605286 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:26.605169 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2aada387-558c97f7cf-5tzj8"] Oct 30 18:50:26.677435 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:26.677404 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:26.788955 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:26.788921 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:26.912898 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:26.912811 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:27.623178 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:27.623126 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f44051-7c85cd7f6f-q4tm9"] Oct 30 18:50:27.623813 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:27.623352 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f44051-7c85cd7f6f-q4tm9" podUID="09ddad39-4b6f-47ef-9c29-1353981233dc" containerName="maas-api" containerID="cri-o://9ee5892a2134b1bb679c68fb9d1618559325f6bb28edafcf812a9aadb465318a" gracePeriod=30 Oct 30 18:50:27.677068 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:27.677033 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:27.999176 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:27.998201 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:28.125573 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:28.125546 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f44051-7c85cd7f6f-q4tm9" Oct 30 18:50:28.174185 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:28.169031 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:28.231964 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:28.231938 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09ddad39-4b6f-47ef-9c29-1353981233dc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09ddad39-4b6f-47ef-9c29-1353981233dc-maas-api-tls\") pod \"09ddad39-4b6f-47ef-9c29-1353981233dc\" (UID: \"09ddad39-4b6f-47ef-9c29-1353981233dc\") " Oct 30 18:50:28.231964 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:28.231976 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09ddad39-4b6f-47ef-9c29-1353981233dc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/09ddad39-4b6f-47ef-9c29-1353981233dc-metrics-certs\") pod \"09ddad39-4b6f-47ef-9c29-1353981233dc\" (UID: \"09ddad39-4b6f-47ef-9c29-1353981233dc\") " Oct 30 18:50:28.232207 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:28.232120 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09ddad39-4b6f-47ef-9c29-1353981233dc-kube-api-access-slmj8\" (UniqueName: \"kubernetes.io/projected/09ddad39-4b6f-47ef-9c29-1353981233dc-kube-api-access-slmj8\") pod \"09ddad39-4b6f-47ef-9c29-1353981233dc\" (UID: \"09ddad39-4b6f-47ef-9c29-1353981233dc\") " Oct 30 18:50:28.232207 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:28.232194 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09ddad39-4b6f-47ef-9c29-1353981233dc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09ddad39-4b6f-47ef-9c29-1353981233dc-openshift-service-ca\") pod \"09ddad39-4b6f-47ef-9c29-1353981233dc\" (UID: \"09ddad39-4b6f-47ef-9c29-1353981233dc\") " Oct 30 18:50:28.232573 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:28.232541 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09ddad39-4b6f-47ef-9c29-1353981233dc-openshift-service-ca" pod "09ddad39-4b6f-47ef-9c29-1353981233dc" (UID: "09ddad39-4b6f-47ef-9c29-1353981233dc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:50:28.232854 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:28.232822 6960 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09ddad39-4b6f-47ef-9c29-1353981233dc-openshift-service-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:50:28.234191 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:28.234166 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09ddad39-4b6f-47ef-9c29-1353981233dc-metrics-certs" pod "09ddad39-4b6f-47ef-9c29-1353981233dc" (UID: "09ddad39-4b6f-47ef-9c29-1353981233dc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:50:28.234290 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:28.234209 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09ddad39-4b6f-47ef-9c29-1353981233dc-maas-api-tls" pod "09ddad39-4b6f-47ef-9c29-1353981233dc" (UID: "09ddad39-4b6f-47ef-9c29-1353981233dc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:50:28.234345 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:28.234313 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09ddad39-4b6f-47ef-9c29-1353981233dc-kube-api-access-slmj8" pod "09ddad39-4b6f-47ef-9c29-1353981233dc" (UID: "09ddad39-4b6f-47ef-9c29-1353981233dc"). InnerVolumeSpecName "kube-api-access-slmj8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:50:28.273009 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:28.272939 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:28.334094 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:28.334063 6960 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09ddad39-4b6f-47ef-9c29-1353981233dc-maas-api-tls\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:50:28.334094 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:28.334097 6960 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09ddad39-4b6f-47ef-9c29-1353981233dc-metrics-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:50:28.334371 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:28.334115 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slmj8\" (UniqueName: \"kubernetes.io/projected/09ddad39-4b6f-47ef-9c29-1353981233dc-kube-api-access-slmj8\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:50:28.587552 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:28.587516 6960 generic.go:356] "Generic (PLEG): container finished" podID="09ddad39-4b6f-47ef-9c29-1353981233dc" containerID="9ee5892a2134b1bb679c68fb9d1618559325f6bb28edafcf812a9aadb465318a" exitCode=0 Oct 30 18:50:28.587727 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:28.587628 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f44051-7c85cd7f6f-q4tm9" Oct 30 18:50:28.594020 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:28.593991 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f44051-7c85cd7f6f-q4tm9" event={"ID":"09ddad39-4b6f-47ef-9c29-1353981233dc","Type":"ContainerDied","Data":"9ee5892a2134b1bb679c68fb9d1618559325f6bb28edafcf812a9aadb465318a"} Oct 30 18:50:28.594186 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:28.594034 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f44051-7c85cd7f6f-q4tm9" event={"ID":"09ddad39-4b6f-47ef-9c29-1353981233dc","Type":"ContainerDied","Data":"3b00d14ad896c63b9fb5bc2861cbc804c9c5e1fa45abcba2dfc0accb24128350"} Oct 30 18:50:28.594186 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:28.594084 6960 scope.go:122] "RemoveContainer" containerID="9ee5892a2134b1bb679c68fb9d1618559325f6bb28edafcf812a9aadb465318a" Oct 30 18:50:28.616557 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:28.616534 6960 scope.go:122] "RemoveContainer" containerID="9ee5892a2134b1bb679c68fb9d1618559325f6bb28edafcf812a9aadb465318a" Oct 30 18:50:28.616953 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:50:28.616928 6960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ee5892a2134b1bb679c68fb9d1618559325f6bb28edafcf812a9aadb465318a\": container with ID starting with 9ee5892a2134b1bb679c68fb9d1618559325f6bb28edafcf812a9aadb465318a not found: ID does not exist" containerID="9ee5892a2134b1bb679c68fb9d1618559325f6bb28edafcf812a9aadb465318a" Oct 30 18:50:28.617063 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:28.616963 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ee5892a2134b1bb679c68fb9d1618559325f6bb28edafcf812a9aadb465318a"} err="failed to get container status \"9ee5892a2134b1bb679c68fb9d1618559325f6bb28edafcf812a9aadb465318a\": rpc error: code = NotFound desc = could not find container \"9ee5892a2134b1bb679c68fb9d1618559325f6bb28edafcf812a9aadb465318a\": container with ID starting with 9ee5892a2134b1bb679c68fb9d1618559325f6bb28edafcf812a9aadb465318a not found: ID does not exist" Oct 30 18:50:28.621307 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:28.621283 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f44051-7c85cd7f6f-q4tm9"] Oct 30 18:50:28.624611 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:28.624587 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f44051-7c85cd7f6f-q4tm9"] Oct 30 18:50:28.711079 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:28.711044 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:29.278926 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:29.278899 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:30.164789 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:30.164760 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:30.367989 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:30.367956 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:30.480495 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:30.480419 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:30.648620 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:30.648405 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09ddad39-4b6f-47ef-9c29-1353981233dc" path="/var/lib/kubelet/pods/09ddad39-4b6f-47ef-9c29-1353981233dc/volumes" Oct 30 18:50:31.256228 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:31.256185 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:31.605601 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:31.605561 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-2aada387-6db6c946b-g28xh" podUID="4a7b452b-b870-4f4b-97af-8bda0657c002" containerName="payload-processing" containerID="cri-o://98e0b39069928ca2a2891d69be94cb6a9fc54fb242f4bd2a5804ca4744f54797" gracePeriod=25 Oct 30 18:50:31.605601 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:31.605583 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-2aada387-558c97f7cf-5tzj8" podUID="71cf7e67-8071-4da9-897c-d791c6b93fbc" containerName="payload-pre-processing" containerID="cri-o://a0eac54c9c25956a0ddeb8647ca2811a8bb68cf6a69cd19f54e7e4c93f4f282d" gracePeriod=25 Oct 30 18:50:32.219436 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:32.219402 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:33.156306 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:33.156274 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:34.043126 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:34.043089 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:34.906549 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:34.906517 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:35.737529 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:35.737486 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:35.928248 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:35.928211 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:36.070178 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:36.070129 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:36.321297 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:36.321222 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-tm4gr"] Oct 30 18:50:36.321518 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:36.321478 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-tm4gr" podUID="4065a953-3883-4ca3-89d2-c81fcbb3444a" containerName="istio-proxy" containerID="cri-o://5952e653f141683cb867ef97c98da801787e1214d2df1a35f1b54b50836b0c76" gracePeriod=30 Oct 30 18:50:36.381355 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:36.381322 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:36.724598 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:36.724505 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-2aada387-558c97f7cf-5tzj8" podUID="71cf7e67-8071-4da9-897c-d791c6b93fbc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:50:36.724598 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:36.724552 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-2aada387-6db6c946b-g28xh" podUID="4a7b452b-b870-4f4b-97af-8bda0657c002" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:50:36.817730 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:36.817699 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:37.022503 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:37.022411 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:37.902376 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:37.902347 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:38.714320 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:38.714279 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:39.005223 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:39.005107 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:39.624072 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:39.624034 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:40.595425 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:40.595387 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:41.411795 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:41.411761 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:41.656900 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:41.656870 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-tm4gr" Oct 30 18:50:41.702531 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:41.702501 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4065a953-3883-4ca3-89d2-c81fcbb3444a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4065a953-3883-4ca3-89d2-c81fcbb3444a-istio-data\") pod \"4065a953-3883-4ca3-89d2-c81fcbb3444a\" (UID: \"4065a953-3883-4ca3-89d2-c81fcbb3444a\") " Oct 30 18:50:41.702819 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:41.702786 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4065a953-3883-4ca3-89d2-c81fcbb3444a-istio-data" pod "4065a953-3883-4ca3-89d2-c81fcbb3444a" (UID: "4065a953-3883-4ca3-89d2-c81fcbb3444a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:50:41.705754 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:41.702886 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4065a953-3883-4ca3-89d2-c81fcbb3444a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4065a953-3883-4ca3-89d2-c81fcbb3444a-credential-socket\") pod \"4065a953-3883-4ca3-89d2-c81fcbb3444a\" (UID: \"4065a953-3883-4ca3-89d2-c81fcbb3444a\") " Oct 30 18:50:41.705754 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:41.702983 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4065a953-3883-4ca3-89d2-c81fcbb3444a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4065a953-3883-4ca3-89d2-c81fcbb3444a-istio-podinfo\") pod \"4065a953-3883-4ca3-89d2-c81fcbb3444a\" (UID: \"4065a953-3883-4ca3-89d2-c81fcbb3444a\") " Oct 30 18:50:41.705754 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:41.703052 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4065a953-3883-4ca3-89d2-c81fcbb3444a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4065a953-3883-4ca3-89d2-c81fcbb3444a-workload-certs\") pod \"4065a953-3883-4ca3-89d2-c81fcbb3444a\" (UID: \"4065a953-3883-4ca3-89d2-c81fcbb3444a\") " Oct 30 18:50:41.705754 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:41.703111 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4065a953-3883-4ca3-89d2-c81fcbb3444a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4065a953-3883-4ca3-89d2-c81fcbb3444a-workload-socket\") pod \"4065a953-3883-4ca3-89d2-c81fcbb3444a\" (UID: \"4065a953-3883-4ca3-89d2-c81fcbb3444a\") " Oct 30 18:50:41.705754 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:41.703227 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4065a953-3883-4ca3-89d2-c81fcbb3444a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4065a953-3883-4ca3-89d2-c81fcbb3444a-istiod-ca-cert\") pod \"4065a953-3883-4ca3-89d2-c81fcbb3444a\" (UID: \"4065a953-3883-4ca3-89d2-c81fcbb3444a\") " Oct 30 18:50:41.705754 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:41.703256 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4065a953-3883-4ca3-89d2-c81fcbb3444a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4065a953-3883-4ca3-89d2-c81fcbb3444a-istio-envoy\") pod \"4065a953-3883-4ca3-89d2-c81fcbb3444a\" (UID: \"4065a953-3883-4ca3-89d2-c81fcbb3444a\") " Oct 30 18:50:41.705754 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:41.703281 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4065a953-3883-4ca3-89d2-c81fcbb3444a-credential-socket" pod "4065a953-3883-4ca3-89d2-c81fcbb3444a" (UID: "4065a953-3883-4ca3-89d2-c81fcbb3444a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:50:41.705754 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:41.703291 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4065a953-3883-4ca3-89d2-c81fcbb3444a-kube-api-access-ckdpp\" (UniqueName: \"kubernetes.io/projected/4065a953-3883-4ca3-89d2-c81fcbb3444a-kube-api-access-ckdpp\") pod \"4065a953-3883-4ca3-89d2-c81fcbb3444a\" (UID: \"4065a953-3883-4ca3-89d2-c81fcbb3444a\") " Oct 30 18:50:41.705754 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:41.703463 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4065a953-3883-4ca3-89d2-c81fcbb3444a-istio-token\" (UniqueName: \"kubernetes.io/projected/4065a953-3883-4ca3-89d2-c81fcbb3444a-istio-token\") pod \"4065a953-3883-4ca3-89d2-c81fcbb3444a\" (UID: \"4065a953-3883-4ca3-89d2-c81fcbb3444a\") " Oct 30 18:50:41.705754 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:41.703799 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4065a953-3883-4ca3-89d2-c81fcbb3444a-workload-socket" pod "4065a953-3883-4ca3-89d2-c81fcbb3444a" (UID: "4065a953-3883-4ca3-89d2-c81fcbb3444a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:50:41.705754 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:41.703982 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4065a953-3883-4ca3-89d2-c81fcbb3444a-workload-certs" pod "4065a953-3883-4ca3-89d2-c81fcbb3444a" (UID: "4065a953-3883-4ca3-89d2-c81fcbb3444a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:50:41.705754 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:41.704431 6960 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4065a953-3883-4ca3-89d2-c81fcbb3444a-workload-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:50:41.705754 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:41.704463 6960 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4065a953-3883-4ca3-89d2-c81fcbb3444a-workload-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:50:41.705754 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:41.704478 6960 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4065a953-3883-4ca3-89d2-c81fcbb3444a-istio-data\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:50:41.705754 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:41.704491 6960 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4065a953-3883-4ca3-89d2-c81fcbb3444a-credential-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:50:41.705754 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:41.704924 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4065a953-3883-4ca3-89d2-c81fcbb3444a-istiod-ca-cert" pod "4065a953-3883-4ca3-89d2-c81fcbb3444a" (UID: "4065a953-3883-4ca3-89d2-c81fcbb3444a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:50:41.715504 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:41.715476 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4065a953-3883-4ca3-89d2-c81fcbb3444a-istio-podinfo" pod "4065a953-3883-4ca3-89d2-c81fcbb3444a" (UID: "4065a953-3883-4ca3-89d2-c81fcbb3444a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:50:41.715594 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:41.715523 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4065a953-3883-4ca3-89d2-c81fcbb3444a-kube-api-access-ckdpp" pod "4065a953-3883-4ca3-89d2-c81fcbb3444a" (UID: "4065a953-3883-4ca3-89d2-c81fcbb3444a"). InnerVolumeSpecName "kube-api-access-ckdpp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:50:41.715594 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:41.715560 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4065a953-3883-4ca3-89d2-c81fcbb3444a-istio-token" pod "4065a953-3883-4ca3-89d2-c81fcbb3444a" (UID: "4065a953-3883-4ca3-89d2-c81fcbb3444a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:50:41.717105 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:41.717080 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4065a953-3883-4ca3-89d2-c81fcbb3444a-istio-envoy" pod "4065a953-3883-4ca3-89d2-c81fcbb3444a" (UID: "4065a953-3883-4ca3-89d2-c81fcbb3444a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:50:41.739451 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:41.739420 6960 generic.go:356] "Generic (PLEG): container finished" podID="4065a953-3883-4ca3-89d2-c81fcbb3444a" containerID="5952e653f141683cb867ef97c98da801787e1214d2df1a35f1b54b50836b0c76" exitCode=0 Oct 30 18:50:41.739544 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:41.739471 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-tm4gr" Oct 30 18:50:41.739544 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:41.739504 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-tm4gr" event={"ID":"4065a953-3883-4ca3-89d2-c81fcbb3444a","Type":"ContainerDied","Data":"5952e653f141683cb867ef97c98da801787e1214d2df1a35f1b54b50836b0c76"} Oct 30 18:50:41.739686 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:41.739574 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-tm4gr" event={"ID":"4065a953-3883-4ca3-89d2-c81fcbb3444a","Type":"ContainerDied","Data":"9a4b9d1d22fc5db0d9e182d0ab33d488fd826c83752dd9811567477e768e674c"} Oct 30 18:50:41.739686 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:41.739620 6960 scope.go:122] "RemoveContainer" containerID="5952e653f141683cb867ef97c98da801787e1214d2df1a35f1b54b50836b0c76" Oct 30 18:50:41.750709 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:41.750694 6960 scope.go:122] "RemoveContainer" containerID="5952e653f141683cb867ef97c98da801787e1214d2df1a35f1b54b50836b0c76" Oct 30 18:50:41.751033 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:50:41.751002 6960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5952e653f141683cb867ef97c98da801787e1214d2df1a35f1b54b50836b0c76\": container with ID starting with 5952e653f141683cb867ef97c98da801787e1214d2df1a35f1b54b50836b0c76 not found: ID does not exist" containerID="5952e653f141683cb867ef97c98da801787e1214d2df1a35f1b54b50836b0c76" Oct 30 18:50:41.751081 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:41.751039 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5952e653f141683cb867ef97c98da801787e1214d2df1a35f1b54b50836b0c76"} err="failed to get container status \"5952e653f141683cb867ef97c98da801787e1214d2df1a35f1b54b50836b0c76\": rpc error: code = NotFound desc = could not find container \"5952e653f141683cb867ef97c98da801787e1214d2df1a35f1b54b50836b0c76\": container with ID starting with 5952e653f141683cb867ef97c98da801787e1214d2df1a35f1b54b50836b0c76 not found: ID does not exist" Oct 30 18:50:41.766416 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:41.766396 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-tm4gr"] Oct 30 18:50:41.773336 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:41.773314 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-f44051-openshift-default-f97b64fbc-tm4gr"] Oct 30 18:50:41.806376 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:41.806346 6960 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4065a953-3883-4ca3-89d2-c81fcbb3444a-istio-podinfo\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:50:41.806524 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:41.806379 6960 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4065a953-3883-4ca3-89d2-c81fcbb3444a-istiod-ca-cert\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:50:41.806524 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:41.806396 6960 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4065a953-3883-4ca3-89d2-c81fcbb3444a-istio-envoy\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:50:41.806524 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:41.806409 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckdpp\" (UniqueName: \"kubernetes.io/projected/4065a953-3883-4ca3-89d2-c81fcbb3444a-kube-api-access-ckdpp\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:50:41.806524 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:41.806425 6960 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4065a953-3883-4ca3-89d2-c81fcbb3444a-istio-token\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:50:42.310306 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:42.310269 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:42.596850 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:42.596734 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4065a953-3883-4ca3-89d2-c81fcbb3444a" path="/var/lib/kubelet/pods/4065a953-3883-4ca3-89d2-c81fcbb3444a/volumes" Oct 30 18:50:43.269493 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:43.269454 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:44.160880 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:44.160846 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:44.890556 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:44.890512 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:45.033003 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:45.032974 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2aada387-577477546b-4fdz6"] Oct 30 18:50:45.033418 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:45.033391 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2aada387-577477546b-4fdz6" podUID="dcdd0a5f-02b9-41be-b090-aa2468942295" containerName="maas-api" containerID="cri-o://c3beaf7efb52400028f8c4e06f44e5f18ed430dab6359d9c85d08f1e6cc07307" gracePeriod=30 Oct 30 18:50:45.104391 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:45.104362 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:45.249516 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:45.241956 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:45.352987 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:45.351539 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2aada387-577477546b-4fdz6" Oct 30 18:50:45.466289 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:45.464407 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:45.476180 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:45.475507 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcdd0a5f-02b9-41be-b090-aa2468942295-kube-api-access-vkrzx\" (UniqueName: \"kubernetes.io/projected/dcdd0a5f-02b9-41be-b090-aa2468942295-kube-api-access-vkrzx\") pod \"dcdd0a5f-02b9-41be-b090-aa2468942295\" (UID: \"dcdd0a5f-02b9-41be-b090-aa2468942295\") " Oct 30 18:50:45.476180 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:45.475631 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dcdd0a5f-02b9-41be-b090-aa2468942295-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dcdd0a5f-02b9-41be-b090-aa2468942295-maas-api-tls\") pod \"dcdd0a5f-02b9-41be-b090-aa2468942295\" (UID: \"dcdd0a5f-02b9-41be-b090-aa2468942295\") " Oct 30 18:50:45.476180 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:45.475730 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dcdd0a5f-02b9-41be-b090-aa2468942295-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcdd0a5f-02b9-41be-b090-aa2468942295-metrics-certs\") pod \"dcdd0a5f-02b9-41be-b090-aa2468942295\" (UID: \"dcdd0a5f-02b9-41be-b090-aa2468942295\") " Oct 30 18:50:45.476180 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:45.475875 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcdd0a5f-02b9-41be-b090-aa2468942295-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcdd0a5f-02b9-41be-b090-aa2468942295-openshift-service-ca\") pod \"dcdd0a5f-02b9-41be-b090-aa2468942295\" (UID: \"dcdd0a5f-02b9-41be-b090-aa2468942295\") " Oct 30 18:50:45.487169 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:45.483924 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcdd0a5f-02b9-41be-b090-aa2468942295-openshift-service-ca" pod "dcdd0a5f-02b9-41be-b090-aa2468942295" (UID: "dcdd0a5f-02b9-41be-b090-aa2468942295"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:50:45.491060 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:45.489541 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcdd0a5f-02b9-41be-b090-aa2468942295-maas-api-tls" pod "dcdd0a5f-02b9-41be-b090-aa2468942295" (UID: "dcdd0a5f-02b9-41be-b090-aa2468942295"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:50:45.502380 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:45.502340 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcdd0a5f-02b9-41be-b090-aa2468942295-kube-api-access-vkrzx" pod "dcdd0a5f-02b9-41be-b090-aa2468942295" (UID: "dcdd0a5f-02b9-41be-b090-aa2468942295"). InnerVolumeSpecName "kube-api-access-vkrzx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:50:45.503288 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:45.503256 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcdd0a5f-02b9-41be-b090-aa2468942295-metrics-certs" pod "dcdd0a5f-02b9-41be-b090-aa2468942295" (UID: "dcdd0a5f-02b9-41be-b090-aa2468942295"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:50:45.578162 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:45.578120 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkrzx\" (UniqueName: \"kubernetes.io/projected/dcdd0a5f-02b9-41be-b090-aa2468942295-kube-api-access-vkrzx\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:50:45.578335 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:45.578201 6960 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dcdd0a5f-02b9-41be-b090-aa2468942295-maas-api-tls\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:50:45.578335 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:45.578214 6960 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcdd0a5f-02b9-41be-b090-aa2468942295-metrics-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:50:45.578335 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:45.578228 6960 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcdd0a5f-02b9-41be-b090-aa2468942295-openshift-service-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:50:45.598936 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:45.598902 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:45.751079 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:45.750982 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:45.769404 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:45.769375 6960 generic.go:356] "Generic (PLEG): container finished" podID="dcdd0a5f-02b9-41be-b090-aa2468942295" containerID="c3beaf7efb52400028f8c4e06f44e5f18ed430dab6359d9c85d08f1e6cc07307" exitCode=0 Oct 30 18:50:45.769584 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:45.769430 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2aada387-577477546b-4fdz6" Oct 30 18:50:45.769584 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:45.769439 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2aada387-577477546b-4fdz6" event={"ID":"dcdd0a5f-02b9-41be-b090-aa2468942295","Type":"ContainerDied","Data":"c3beaf7efb52400028f8c4e06f44e5f18ed430dab6359d9c85d08f1e6cc07307"} Oct 30 18:50:45.769584 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:45.769480 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2aada387-577477546b-4fdz6" event={"ID":"dcdd0a5f-02b9-41be-b090-aa2468942295","Type":"ContainerDied","Data":"37c428beba8a1cbb69fb8c96a88cbc1d98ae27d329013f87762b1f8f2e9dfc29"} Oct 30 18:50:45.769584 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:45.769500 6960 scope.go:122] "RemoveContainer" containerID="c3beaf7efb52400028f8c4e06f44e5f18ed430dab6359d9c85d08f1e6cc07307" Oct 30 18:50:45.783540 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:45.783519 6960 scope.go:122] "RemoveContainer" containerID="c3beaf7efb52400028f8c4e06f44e5f18ed430dab6359d9c85d08f1e6cc07307" Oct 30 18:50:45.783925 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:50:45.783898 6960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3beaf7efb52400028f8c4e06f44e5f18ed430dab6359d9c85d08f1e6cc07307\": container with ID starting with c3beaf7efb52400028f8c4e06f44e5f18ed430dab6359d9c85d08f1e6cc07307 not found: ID does not exist" containerID="c3beaf7efb52400028f8c4e06f44e5f18ed430dab6359d9c85d08f1e6cc07307" Oct 30 18:50:45.784016 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:45.783940 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3beaf7efb52400028f8c4e06f44e5f18ed430dab6359d9c85d08f1e6cc07307"} err="failed to get container status \"c3beaf7efb52400028f8c4e06f44e5f18ed430dab6359d9c85d08f1e6cc07307\": rpc error: code = NotFound desc = could not find container \"c3beaf7efb52400028f8c4e06f44e5f18ed430dab6359d9c85d08f1e6cc07307\": container with ID starting with c3beaf7efb52400028f8c4e06f44e5f18ed430dab6359d9c85d08f1e6cc07307 not found: ID does not exist" Oct 30 18:50:45.801436 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:45.801407 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2aada387-577477546b-4fdz6"] Oct 30 18:50:45.805684 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:45.805661 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2aada387-577477546b-4fdz6"] Oct 30 18:50:45.870714 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:45.870685 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:46.593612 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:46.593584 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcdd0a5f-02b9-41be-b090-aa2468942295" path="/var/lib/kubelet/pods/dcdd0a5f-02b9-41be-b090-aa2468942295/volumes" Oct 30 18:50:46.618914 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:46.618887 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:46.724168 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:46.724115 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-2aada387-558c97f7cf-5tzj8" podUID="71cf7e67-8071-4da9-897c-d791c6b93fbc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:50:46.724402 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:46.724128 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-2aada387-6db6c946b-g28xh" podUID="4a7b452b-b870-4f4b-97af-8bda0657c002" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:50:47.654171 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:47.653925 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:48.711255 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:48.710511 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:49.514808 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:49.514759 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:50.205744 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:50.205700 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:50.322060 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:50.322030 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:50.520926 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:50.520851 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:50.769592 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:50.769557 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:51.312843 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:51.312804 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:51.468230 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:51.468192 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:51.827181 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:51.827130 6960 generic.go:356] "Generic (PLEG): container finished" podID="71cf7e67-8071-4da9-897c-d791c6b93fbc" containerID="a0eac54c9c25956a0ddeb8647ca2811a8bb68cf6a69cd19f54e7e4c93f4f282d" exitCode=0 Oct 30 18:50:51.827287 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:51.827177 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2aada387-558c97f7cf-5tzj8" event={"ID":"71cf7e67-8071-4da9-897c-d791c6b93fbc","Type":"ContainerDied","Data":"a0eac54c9c25956a0ddeb8647ca2811a8bb68cf6a69cd19f54e7e4c93f4f282d"} Oct 30 18:50:51.839808 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:51.839775 6960 generic.go:356] "Generic (PLEG): container finished" podID="4a7b452b-b870-4f4b-97af-8bda0657c002" containerID="98e0b39069928ca2a2891d69be94cb6a9fc54fb242f4bd2a5804ca4744f54797" exitCode=0 Oct 30 18:50:51.839936 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:51.839921 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2aada387-6db6c946b-g28xh" event={"ID":"4a7b452b-b870-4f4b-97af-8bda0657c002","Type":"ContainerDied","Data":"98e0b39069928ca2a2891d69be94cb6a9fc54fb242f4bd2a5804ca4744f54797"} Oct 30 18:50:51.905874 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:51.905845 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2aada387-6db6c946b-g28xh" Oct 30 18:50:51.959518 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:51.959430 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:51.987705 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:51.983656 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2aada387-558c97f7cf-5tzj8" Oct 30 18:50:51.988456 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:51.988430 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a7b452b-b870-4f4b-97af-8bda0657c002-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a7b452b-b870-4f4b-97af-8bda0657c002-plugins-config-volume\") pod \"4a7b452b-b870-4f4b-97af-8bda0657c002\" (UID: \"4a7b452b-b870-4f4b-97af-8bda0657c002\") " Oct 30 18:50:51.988614 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:51.988472 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a7b452b-b870-4f4b-97af-8bda0657c002-kube-api-access-lgvpj\" (UniqueName: \"kubernetes.io/projected/4a7b452b-b870-4f4b-97af-8bda0657c002-kube-api-access-lgvpj\") pod \"4a7b452b-b870-4f4b-97af-8bda0657c002\" (UID: \"4a7b452b-b870-4f4b-97af-8bda0657c002\") " Oct 30 18:50:51.989062 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:51.989030 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a7b452b-b870-4f4b-97af-8bda0657c002-plugins-config-volume" pod "4a7b452b-b870-4f4b-97af-8bda0657c002" (UID: "4a7b452b-b870-4f4b-97af-8bda0657c002"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:50:51.989373 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:51.989354 6960 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a7b452b-b870-4f4b-97af-8bda0657c002-plugins-config-volume\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:50:51.991762 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:51.991727 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a7b452b-b870-4f4b-97af-8bda0657c002-kube-api-access-lgvpj" pod "4a7b452b-b870-4f4b-97af-8bda0657c002" (UID: "4a7b452b-b870-4f4b-97af-8bda0657c002"). InnerVolumeSpecName "kube-api-access-lgvpj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:50:52.090813 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:52.090785 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71cf7e67-8071-4da9-897c-d791c6b93fbc-kube-api-access-wvb86\" (UniqueName: \"kubernetes.io/projected/71cf7e67-8071-4da9-897c-d791c6b93fbc-kube-api-access-wvb86\") pod \"71cf7e67-8071-4da9-897c-d791c6b93fbc\" (UID: \"71cf7e67-8071-4da9-897c-d791c6b93fbc\") " Oct 30 18:50:52.090999 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:52.090912 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71cf7e67-8071-4da9-897c-d791c6b93fbc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71cf7e67-8071-4da9-897c-d791c6b93fbc-plugins-config-volume\") pod \"71cf7e67-8071-4da9-897c-d791c6b93fbc\" (UID: \"71cf7e67-8071-4da9-897c-d791c6b93fbc\") " Oct 30 18:50:52.091317 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:52.091293 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71cf7e67-8071-4da9-897c-d791c6b93fbc-plugins-config-volume" pod "71cf7e67-8071-4da9-897c-d791c6b93fbc" (UID: "71cf7e67-8071-4da9-897c-d791c6b93fbc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:50:52.091430 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:52.091417 6960 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71cf7e67-8071-4da9-897c-d791c6b93fbc-plugins-config-volume\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:50:52.091477 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:52.091433 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgvpj\" (UniqueName: \"kubernetes.io/projected/4a7b452b-b870-4f4b-97af-8bda0657c002-kube-api-access-lgvpj\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:50:52.093117 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:52.093095 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71cf7e67-8071-4da9-897c-d791c6b93fbc-kube-api-access-wvb86" pod "71cf7e67-8071-4da9-897c-d791c6b93fbc" (UID: "71cf7e67-8071-4da9-897c-d791c6b93fbc"). InnerVolumeSpecName "kube-api-access-wvb86". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:50:52.193139 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:52.193062 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvb86\" (UniqueName: \"kubernetes.io/projected/71cf7e67-8071-4da9-897c-d791c6b93fbc-kube-api-access-wvb86\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:50:52.847480 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:52.847458 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2aada387-558c97f7cf-5tzj8" Oct 30 18:50:52.847799 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:52.847458 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2aada387-558c97f7cf-5tzj8" event={"ID":"71cf7e67-8071-4da9-897c-d791c6b93fbc","Type":"ContainerDied","Data":"4945373a9bb26add8abdfc9da0cf844328d990a6135f60257ee6cdbfd4047142"} Oct 30 18:50:52.847799 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:52.847596 6960 scope.go:122] "RemoveContainer" containerID="a0eac54c9c25956a0ddeb8647ca2811a8bb68cf6a69cd19f54e7e4c93f4f282d" Oct 30 18:50:52.850039 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:52.850015 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-2aada387-6db6c946b-g28xh" Oct 30 18:50:52.850705 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:52.849928 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-2aada387-6db6c946b-g28xh" event={"ID":"4a7b452b-b870-4f4b-97af-8bda0657c002","Type":"ContainerDied","Data":"82794cdc58681006ed67c3074462e05015400a44964138af3cef67c3a475924a"} Oct 30 18:50:52.857552 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:52.857531 6960 scope.go:122] "RemoveContainer" containerID="98e0b39069928ca2a2891d69be94cb6a9fc54fb242f4bd2a5804ca4744f54797" Oct 30 18:50:52.879554 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:52.879531 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2aada387-6db6c946b-g28xh"] Oct 30 18:50:52.886031 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:52.886009 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-2aada387-6db6c946b-g28xh"] Oct 30 18:50:52.897187 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:52.897166 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2aada387-558c97f7cf-5tzj8"] Oct 30 18:50:52.903353 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:52.903336 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2aada387-558c97f7cf-5tzj8"] Oct 30 18:50:52.970732 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:52.969734 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:53.934666 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:50:53.934632 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 18:50:54.039006 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:54.038962 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:54.593632 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:54.593602 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a7b452b-b870-4f4b-97af-8bda0657c002" path="/var/lib/kubelet/pods/4a7b452b-b870-4f4b-97af-8bda0657c002/volumes" Oct 30 18:50:54.594035 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:54.594022 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71cf7e67-8071-4da9-897c-d791c6b93fbc" path="/var/lib/kubelet/pods/71cf7e67-8071-4da9-897c-d791c6b93fbc/volumes" Oct 30 18:50:54.763275 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:54.763241 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:55.053677 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:55.053038 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:55.305227 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:55.300351 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:56.091219 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:56.088651 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:57.128223 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:57.128189 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:58.289494 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:58.289464 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:58.560715 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:58.560684 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:58.714971 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:58.714938 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:59.053982 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:59.053901 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:59.202463 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:59.202272 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:59.455875 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:50:59.455846 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:00.455236 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:00.455199 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:00.738936 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:00.738825 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:00.882663 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:00.882625 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:01.139838 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:01.139785 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:01.296473 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:01.296440 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:01.409763 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:01.409684 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:01.522027 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:01.521965 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:01.698530 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:01.698448 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:01.903878 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:01.903846 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:02.505650 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:02.505612 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:02.715200 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:02.715162 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:03.089123 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:03.089082 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:03.264527 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:03.264477 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:03.520194 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:03.520077 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:04.527785 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:04.527750 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:04.691467 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:04.691430 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:05.816323 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:05.816290 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:07.213970 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:07.213939 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:07.350985 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:07.350949 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:07.838625 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:07.838591 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:07.997666 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:07.997624 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:08.193043 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:08.192963 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:08.696843 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:08.696813 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:09.004775 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:09.004704 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:09.123324 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:09.123293 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:09.787777 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:09.787743 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:10.908313 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:10.908278 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:11.715041 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:11.715005 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:12.625856 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:12.625821 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:12.735077 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:12.735039 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:13.599839 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:13.599797 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:13.707084 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:13.707053 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:13.824621 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:13.824588 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:14.729406 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:14.729371 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:15.598666 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:15.598634 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:16.562360 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:16.562329 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5ad713-798d856f68-xqm68"] Oct 30 18:51:16.562941 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:16.562916 6960 cpu_manager.go:414] "containerMap: removing container" podUID="4065a953-3883-4ca3-89d2-c81fcbb3444a" containerName="istio-proxy" Oct 30 18:51:16.562941 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:16.562932 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4065a953-3883-4ca3-89d2-c81fcbb3444a" containerName="istio-proxy" Oct 30 18:51:16.563106 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:16.562949 6960 cpu_manager.go:414] "containerMap: removing container" podUID="c6900142-912c-4ac1-a0d6-cb7e97e0fad5" containerName="payload-pre-processing" Oct 30 18:51:16.563106 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:16.562955 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6900142-912c-4ac1-a0d6-cb7e97e0fad5" containerName="payload-pre-processing" Oct 30 18:51:16.563106 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:16.562969 6960 cpu_manager.go:414] "containerMap: removing container" podUID="e66b2d2a-c6e7-4300-b37a-d8e82408fa35" containerName="payload-processing" Oct 30 18:51:16.563106 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:16.562977 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e66b2d2a-c6e7-4300-b37a-d8e82408fa35" containerName="payload-processing" Oct 30 18:51:16.563106 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:16.562987 6960 cpu_manager.go:414] "containerMap: removing container" podUID="09ddad39-4b6f-47ef-9c29-1353981233dc" containerName="maas-api" Oct 30 18:51:16.563106 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:16.562994 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09ddad39-4b6f-47ef-9c29-1353981233dc" containerName="maas-api" Oct 30 18:51:16.563106 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:16.563012 6960 cpu_manager.go:414] "containerMap: removing container" podUID="fb28fce6-3142-4677-9cec-a018e5764832" containerName="maas-api" Oct 30 18:51:16.563106 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:16.563017 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb28fce6-3142-4677-9cec-a018e5764832" containerName="maas-api" Oct 30 18:51:16.563106 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:16.563037 6960 cpu_manager.go:414] "containerMap: removing container" podUID="71cf7e67-8071-4da9-897c-d791c6b93fbc" containerName="payload-pre-processing" Oct 30 18:51:16.563106 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:16.563043 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71cf7e67-8071-4da9-897c-d791c6b93fbc" containerName="payload-pre-processing" Oct 30 18:51:16.563106 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:16.563050 6960 cpu_manager.go:414] "containerMap: removing container" podUID="4a7b452b-b870-4f4b-97af-8bda0657c002" containerName="payload-processing" Oct 30 18:51:16.563106 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:16.563055 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a7b452b-b870-4f4b-97af-8bda0657c002" containerName="payload-processing" Oct 30 18:51:16.563106 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:16.563076 6960 cpu_manager.go:414] "containerMap: removing container" podUID="dcdd0a5f-02b9-41be-b090-aa2468942295" containerName="maas-api" Oct 30 18:51:16.563106 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:16.563084 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcdd0a5f-02b9-41be-b090-aa2468942295" containerName="maas-api" Oct 30 18:51:16.563709 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:16.563221 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="71cf7e67-8071-4da9-897c-d791c6b93fbc" containerName="payload-pre-processing" Oct 30 18:51:16.563709 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:16.563235 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a7b452b-b870-4f4b-97af-8bda0657c002" containerName="payload-processing" Oct 30 18:51:16.563709 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:16.563253 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcdd0a5f-02b9-41be-b090-aa2468942295" containerName="maas-api" Oct 30 18:51:16.563709 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:16.563264 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="4065a953-3883-4ca3-89d2-c81fcbb3444a" containerName="istio-proxy" Oct 30 18:51:16.563709 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:16.563271 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6900142-912c-4ac1-a0d6-cb7e97e0fad5" containerName="payload-pre-processing" Oct 30 18:51:16.563709 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:16.563277 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="e66b2d2a-c6e7-4300-b37a-d8e82408fa35" containerName="payload-processing" Oct 30 18:51:16.563709 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:16.563286 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="09ddad39-4b6f-47ef-9c29-1353981233dc" containerName="maas-api" Oct 30 18:51:16.563709 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:16.563294 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb28fce6-3142-4677-9cec-a018e5764832" containerName="maas-api" Oct 30 18:51:16.566829 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:16.566811 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5ad713-798d856f68-xqm68" Oct 30 18:51:16.575025 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:16.574995 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-5ad713-serving-cert\"" Oct 30 18:51:16.580118 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:16.580091 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5ad713-798d856f68-xqm68"] Oct 30 18:51:16.651687 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:16.651646 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:16.728121 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:16.728085 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed0b8197-ddea-4c33-93c9-286d42ea7322-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-5ad713-798d856f68-xqm68\" (UID: \"ed0b8197-ddea-4c33-93c9-286d42ea7322\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5ad713-798d856f68-xqm68" Oct 30 18:51:16.728319 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:16.728240 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed0b8197-ddea-4c33-93c9-286d42ea7322-maas-api-tls\") pod \"maas-api-e2e-ipp-a-5ad713-798d856f68-xqm68\" (UID: \"ed0b8197-ddea-4c33-93c9-286d42ea7322\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5ad713-798d856f68-xqm68" Oct 30 18:51:16.728447 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:16.728426 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed0b8197-ddea-4c33-93c9-286d42ea7322-metrics-certs\") pod \"maas-api-e2e-ipp-a-5ad713-798d856f68-xqm68\" (UID: \"ed0b8197-ddea-4c33-93c9-286d42ea7322\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5ad713-798d856f68-xqm68" Oct 30 18:51:16.728637 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:16.728615 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq7t4\" (UniqueName: \"kubernetes.io/projected/ed0b8197-ddea-4c33-93c9-286d42ea7322-kube-api-access-tq7t4\") pod \"maas-api-e2e-ipp-a-5ad713-798d856f68-xqm68\" (UID: \"ed0b8197-ddea-4c33-93c9-286d42ea7322\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5ad713-798d856f68-xqm68" Oct 30 18:51:16.759566 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:16.759538 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:16.830202 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:16.830076 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed0b8197-ddea-4c33-93c9-286d42ea7322-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-5ad713-798d856f68-xqm68\" (UID: \"ed0b8197-ddea-4c33-93c9-286d42ea7322\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5ad713-798d856f68-xqm68" Oct 30 18:51:16.830202 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:16.830135 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed0b8197-ddea-4c33-93c9-286d42ea7322-maas-api-tls\") pod \"maas-api-e2e-ipp-a-5ad713-798d856f68-xqm68\" (UID: \"ed0b8197-ddea-4c33-93c9-286d42ea7322\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5ad713-798d856f68-xqm68" Oct 30 18:51:16.830202 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:16.830190 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed0b8197-ddea-4c33-93c9-286d42ea7322-metrics-certs\") pod \"maas-api-e2e-ipp-a-5ad713-798d856f68-xqm68\" (UID: \"ed0b8197-ddea-4c33-93c9-286d42ea7322\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5ad713-798d856f68-xqm68" Oct 30 18:51:16.830488 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:16.830293 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq7t4\" (UniqueName: \"kubernetes.io/projected/ed0b8197-ddea-4c33-93c9-286d42ea7322-kube-api-access-tq7t4\") pod \"maas-api-e2e-ipp-a-5ad713-798d856f68-xqm68\" (UID: \"ed0b8197-ddea-4c33-93c9-286d42ea7322\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5ad713-798d856f68-xqm68" Oct 30 18:51:16.830911 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:16.830884 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed0b8197-ddea-4c33-93c9-286d42ea7322-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-5ad713-798d856f68-xqm68\" (UID: \"ed0b8197-ddea-4c33-93c9-286d42ea7322\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5ad713-798d856f68-xqm68" Oct 30 18:51:16.832999 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:16.832970 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed0b8197-ddea-4c33-93c9-286d42ea7322-metrics-certs\") pod \"maas-api-e2e-ipp-a-5ad713-798d856f68-xqm68\" (UID: \"ed0b8197-ddea-4c33-93c9-286d42ea7322\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5ad713-798d856f68-xqm68" Oct 30 18:51:16.833364 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:16.833342 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed0b8197-ddea-4c33-93c9-286d42ea7322-maas-api-tls\") pod \"maas-api-e2e-ipp-a-5ad713-798d856f68-xqm68\" (UID: \"ed0b8197-ddea-4c33-93c9-286d42ea7322\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5ad713-798d856f68-xqm68" Oct 30 18:51:16.842488 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:16.842463 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq7t4\" (UniqueName: \"kubernetes.io/projected/ed0b8197-ddea-4c33-93c9-286d42ea7322-kube-api-access-tq7t4\") pod \"maas-api-e2e-ipp-a-5ad713-798d856f68-xqm68\" (UID: \"ed0b8197-ddea-4c33-93c9-286d42ea7322\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5ad713-798d856f68-xqm68" Oct 30 18:51:16.872656 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:16.872621 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:16.886434 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:16.886389 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5ad713-798d856f68-xqm68" Oct 30 18:51:17.194467 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:17.194436 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5ad713-798d856f68-xqm68"] Oct 30 18:51:17.198334 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:17.197234 6960 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 18:51:17.926014 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:17.925929 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:18.050680 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:18.050643 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5ad713-798d856f68-xqm68" event={"ID":"ed0b8197-ddea-4c33-93c9-286d42ea7322","Type":"ContainerStarted","Data":"9c72f4f1e281421b0f7e969ec0a6fb9cca5d6fe4a34eb9573d673ad61e1d0b8e"} Oct 30 18:51:18.050893 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:18.050706 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5ad713-798d856f68-xqm68" Oct 30 18:51:18.050893 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:18.050727 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5ad713-798d856f68-xqm68" event={"ID":"ed0b8197-ddea-4c33-93c9-286d42ea7322","Type":"ContainerStarted","Data":"c263dc91a5f86fe33df3edcdaac3e593295fc81e25d868ce41668e204771f018"} Oct 30 18:51:18.076373 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:18.076312 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5ad713-798d856f68-xqm68" podStartSLOduration=1.6239516219999999 podStartE2EDuration="2.076293698s" totalImagesPullingTime="452.342076ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:51:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:51:18.069813775 +0000 UTC m=+1045.598844913" watchObservedRunningTime="2026-10-30 18:51:18.076293698 +0000 UTC m=+1045.605324838" Oct 30 18:51:18.917599 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:18.917564 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:19.617130 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:19.617094 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:20.568647 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:20.568606 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:21.411777 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:21.411746 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:22.353490 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:22.353460 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:23.106128 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:23.106095 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:23.293280 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:23.293247 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:23.933235 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:51:23.933200 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 18:51:24.062603 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:24.062576 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5ad713-798d856f68-xqm68" Oct 30 18:51:24.465013 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:24.464972 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:25.719313 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:25.719278 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:25.962786 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:25.962756 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:26.092674 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:26.092640 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:26.253682 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:26.253649 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:26.401954 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:26.401870 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:27.058690 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:27.058657 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:28.150403 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:28.150366 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:29.008562 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:29.008522 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:30.098572 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:30.098532 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:30.843741 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:30.843706 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:31.619948 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:31.619897 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:32.410131 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:32.410100 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:32.876252 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:32.876208 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:32.989320 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:32.989289 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:33.094740 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:33.094702 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:33.232537 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:33.232440 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:33.719919 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:33.719883 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:34.447057 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:34.447018 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:34.686232 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:34.686161 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:34.845279 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:34.845244 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:34.967661 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:34.967621 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:35.095382 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:35.095273 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:35.443300 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:35.443204 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:36.339964 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:36.339931 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:36.925488 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:36.925450 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:37.039760 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:37.039729 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:37.527418 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:37.527347 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:38.547445 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:38.547412 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:39.324835 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:39.324804 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:40.181218 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:40.181187 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:41.161933 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:41.161895 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:42.012251 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:42.012207 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:43.019499 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:43.019467 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:44.117851 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:44.117813 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:44.250430 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:44.250393 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:44.717188 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:44.717132 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:45.211542 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:45.211502 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:46.208332 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:46.208301 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:47.047676 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:47.047642 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:47.899989 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:47.899955 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:49.015000 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:49.014963 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:49.697478 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:49.697445 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-2aada387/e2e-lifecycle-model-2aada387-kserve-dcfbdc957-48c2n"] Oct 30 18:51:49.758618 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:49.758578 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:50.098115 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:50.098080 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:50.926198 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:50.926161 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:51.265061 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:51.264980 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:52.013453 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:52.013419 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:52.711174 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:52.711120 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:52.862159 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:52.861996 6960 scope.go:122] "RemoveContainer" containerID="820c9ee17cf20f5cd7bea3349236ec71c9747ebc77683e654ad364ffb2b76ff8" Oct 30 18:51:53.096637 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:53.096598 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:53.224592 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:53.224561 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:53.373896 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:53.373800 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:53.935973 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:51:53.935937 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 18:51:53.999100 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:53.999074 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:54.857841 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:54.857807 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:55.110194 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:55.110065 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:55.475193 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:55.475063 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:55.817695 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:55.817661 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:56.364759 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:56.364717 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:57.353254 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:57.353192 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:57.544596 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:57.544542 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:58.152574 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:58.152533 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:58.409447 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:58.409345 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:58.666757 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:58.665063 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:58.793698 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:58.793651 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:59.476798 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:59.476760 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:59.658888 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:59.658850 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:59.941947 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:51:59.941906 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:01.010237 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:01.010204 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:01.268557 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:01.268464 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:01.455706 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:01.455679 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:01.615087 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:01.615055 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:01.813301 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:01.813269 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:02.675217 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:02.675178 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:02.818202 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:02.818128 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:02.978220 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:02.969619 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:04.087037 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:04.086959 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:04.306767 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:04.306736 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-45f649-849bdd96c9-mwhwd"] Oct 30 18:52:04.312772 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:04.312723 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45f649-849bdd96c9-mwhwd" Oct 30 18:52:04.315682 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:52:04.315645 6960 status_manager.go:1045] "Failed to get status for pod" err="pods \"payload-pre-processing-e2e-ipp-b-45f649-849bdd96c9-mwhwd\" is forbidden: User \"system:node:ip-10-0-140-163.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-140-163.us-east-2.compute.internal' and this object" podUID="09fe7720-3d9d-4327-a6a5-b94bbcff94ca" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45f649-849bdd96c9-mwhwd" Oct 30 18:52:04.320990 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:52:04.320936 6960 reflector.go:204] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"payload-processing-plugins-e2e-ipp-b-45f649\" is forbidden: User \"system:node:ip-10-0-140-163.us-east-2.compute.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-140-163.us-east-2.compute.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-45f649\"" type="*v1.ConfigMap" Oct 30 18:52:04.341362 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:52:04.341039 6960 reflector.go:204] "Failed to watch" err="failed to list *v1.Secret: secrets \"payload-processing-e2e-ipp-b-45f649-dockercfg-jndg9\" is forbidden: User \"system:node:ip-10-0-140-163.us-east-2.compute.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-140-163.us-east-2.compute.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-45f649-dockercfg-jndg9\"" type="*v1.Secret" Oct 30 18:52:04.370238 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:04.370206 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-45f649-7564cc6b9d-rvfdp"] Oct 30 18:52:04.389171 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:04.375450 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-45f649-7564cc6b9d-rvfdp" Oct 30 18:52:04.389171 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:04.377620 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09fe7720-3d9d-4327-a6a5-b94bbcff94ca-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-45f649-849bdd96c9-mwhwd\" (UID: \"09fe7720-3d9d-4327-a6a5-b94bbcff94ca\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45f649-849bdd96c9-mwhwd" Oct 30 18:52:04.389171 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:04.378634 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7426960-9456-44cc-9de3-13158003510b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-45f649-7564cc6b9d-rvfdp\" (UID: \"a7426960-9456-44cc-9de3-13158003510b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-45f649-7564cc6b9d-rvfdp" Oct 30 18:52:04.389171 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:04.378693 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mk4w\" (UniqueName: \"kubernetes.io/projected/a7426960-9456-44cc-9de3-13158003510b-kube-api-access-8mk4w\") pod \"payload-processing-e2e-ipp-b-45f649-7564cc6b9d-rvfdp\" (UID: \"a7426960-9456-44cc-9de3-13158003510b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-45f649-7564cc6b9d-rvfdp" Oct 30 18:52:04.389171 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:04.378771 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8x56g\" (UniqueName: \"kubernetes.io/projected/09fe7720-3d9d-4327-a6a5-b94bbcff94ca-kube-api-access-8x56g\") pod \"payload-pre-processing-e2e-ipp-b-45f649-849bdd96c9-mwhwd\" (UID: \"09fe7720-3d9d-4327-a6a5-b94bbcff94ca\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45f649-849bdd96c9-mwhwd" Oct 30 18:52:04.415788 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:04.415206 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-45f649-849bdd96c9-mwhwd"] Oct 30 18:52:04.448377 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:04.436364 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-45f649-7564cc6b9d-rvfdp"] Oct 30 18:52:04.473172 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:04.473120 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:04.498206 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:04.492611 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7426960-9456-44cc-9de3-13158003510b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-45f649-7564cc6b9d-rvfdp\" (UID: \"a7426960-9456-44cc-9de3-13158003510b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-45f649-7564cc6b9d-rvfdp" Oct 30 18:52:04.498206 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:04.492665 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mk4w\" (UniqueName: \"kubernetes.io/projected/a7426960-9456-44cc-9de3-13158003510b-kube-api-access-8mk4w\") pod \"payload-processing-e2e-ipp-b-45f649-7564cc6b9d-rvfdp\" (UID: \"a7426960-9456-44cc-9de3-13158003510b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-45f649-7564cc6b9d-rvfdp" Oct 30 18:52:04.498206 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:04.492756 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x56g\" (UniqueName: \"kubernetes.io/projected/09fe7720-3d9d-4327-a6a5-b94bbcff94ca-kube-api-access-8x56g\") pod \"payload-pre-processing-e2e-ipp-b-45f649-849bdd96c9-mwhwd\" (UID: \"09fe7720-3d9d-4327-a6a5-b94bbcff94ca\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45f649-849bdd96c9-mwhwd" Oct 30 18:52:04.498206 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:04.492882 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09fe7720-3d9d-4327-a6a5-b94bbcff94ca-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-45f649-849bdd96c9-mwhwd\" (UID: \"09fe7720-3d9d-4327-a6a5-b94bbcff94ca\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45f649-849bdd96c9-mwhwd" Oct 30 18:52:04.519478 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:04.519015 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8x56g\" (UniqueName: \"kubernetes.io/projected/09fe7720-3d9d-4327-a6a5-b94bbcff94ca-kube-api-access-8x56g\") pod \"payload-pre-processing-e2e-ipp-b-45f649-849bdd96c9-mwhwd\" (UID: \"09fe7720-3d9d-4327-a6a5-b94bbcff94ca\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45f649-849bdd96c9-mwhwd" Oct 30 18:52:04.526581 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:04.526546 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mk4w\" (UniqueName: \"kubernetes.io/projected/a7426960-9456-44cc-9de3-13158003510b-kube-api-access-8mk4w\") pod \"payload-processing-e2e-ipp-b-45f649-7564cc6b9d-rvfdp\" (UID: \"a7426960-9456-44cc-9de3-13158003510b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-45f649-7564cc6b9d-rvfdp" Oct 30 18:52:04.618249 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:04.618121 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:04.732522 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:04.732456 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-2aada387/e2e-lifecycle-model-2aada387-kserve-dcfbdc957-48c2n" podUID="8c95874c-5d7b-4aeb-b968-06af27377c8e" containerName="main" containerID="cri-o://9bb352a012907d22139064c0a28767116c4c0d317a5ca9ba5567d79abb6a6e0b" gracePeriod=45 Oct 30 18:52:04.808458 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:04.808425 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-2aada387/e2e-lifecycle-model-2aada387-kserve-dcfbdc957-48c2n" podUID="8c95874c-5d7b-4aeb-b968-06af27377c8e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": dial tcp 10.134.0.78:8000: connect: connection refused" Oct 30 18:52:05.018658 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:05.018634 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-2aada387/e2e-lifecycle-model-2aada387-kserve-dcfbdc957-48c2n" Oct 30 18:52:05.100424 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:05.100388 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c95874c-5d7b-4aeb-b968-06af27377c8e-kube-api-access-btdw9\" (UniqueName: \"kubernetes.io/projected/8c95874c-5d7b-4aeb-b968-06af27377c8e-kube-api-access-btdw9\") pod \"8c95874c-5d7b-4aeb-b968-06af27377c8e\" (UID: \"8c95874c-5d7b-4aeb-b968-06af27377c8e\") " Oct 30 18:52:05.100895 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:05.100526 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c95874c-5d7b-4aeb-b968-06af27377c8e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c95874c-5d7b-4aeb-b968-06af27377c8e-dshm\") pod \"8c95874c-5d7b-4aeb-b968-06af27377c8e\" (UID: \"8c95874c-5d7b-4aeb-b968-06af27377c8e\") " Oct 30 18:52:05.100895 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:05.100602 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c95874c-5d7b-4aeb-b968-06af27377c8e-home\" (UniqueName: \"kubernetes.io/empty-dir/8c95874c-5d7b-4aeb-b968-06af27377c8e-home\") pod \"8c95874c-5d7b-4aeb-b968-06af27377c8e\" (UID: \"8c95874c-5d7b-4aeb-b968-06af27377c8e\") " Oct 30 18:52:05.100895 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:05.100739 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c95874c-5d7b-4aeb-b968-06af27377c8e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c95874c-5d7b-4aeb-b968-06af27377c8e-model-cache\") pod \"8c95874c-5d7b-4aeb-b968-06af27377c8e\" (UID: \"8c95874c-5d7b-4aeb-b968-06af27377c8e\") " Oct 30 18:52:05.100895 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:05.100798 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c95874c-5d7b-4aeb-b968-06af27377c8e-tls-certs\" (UniqueName: \"kubernetes.io/secret/8c95874c-5d7b-4aeb-b968-06af27377c8e-tls-certs\") pod \"8c95874c-5d7b-4aeb-b968-06af27377c8e\" (UID: \"8c95874c-5d7b-4aeb-b968-06af27377c8e\") " Oct 30 18:52:05.100895 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:05.100831 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c95874c-5d7b-4aeb-b968-06af27377c8e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c95874c-5d7b-4aeb-b968-06af27377c8e-tmp-dir\") pod \"8c95874c-5d7b-4aeb-b968-06af27377c8e\" (UID: \"8c95874c-5d7b-4aeb-b968-06af27377c8e\") " Oct 30 18:52:05.101178 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:05.100900 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c95874c-5d7b-4aeb-b968-06af27377c8e-home" pod "8c95874c-5d7b-4aeb-b968-06af27377c8e" (UID: "8c95874c-5d7b-4aeb-b968-06af27377c8e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:52:05.101178 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:05.101037 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c95874c-5d7b-4aeb-b968-06af27377c8e-model-cache" pod "8c95874c-5d7b-4aeb-b968-06af27377c8e" (UID: "8c95874c-5d7b-4aeb-b968-06af27377c8e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:52:05.101403 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:05.101385 6960 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c95874c-5d7b-4aeb-b968-06af27377c8e-home\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:52:05.101473 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:05.101409 6960 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c95874c-5d7b-4aeb-b968-06af27377c8e-model-cache\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:52:05.101520 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:05.101489 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c95874c-5d7b-4aeb-b968-06af27377c8e-tmp-dir" pod "8c95874c-5d7b-4aeb-b968-06af27377c8e" (UID: "8c95874c-5d7b-4aeb-b968-06af27377c8e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:52:05.103093 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:05.103070 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c95874c-5d7b-4aeb-b968-06af27377c8e-dshm" pod "8c95874c-5d7b-4aeb-b968-06af27377c8e" (UID: "8c95874c-5d7b-4aeb-b968-06af27377c8e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:52:05.103396 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:05.103374 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c95874c-5d7b-4aeb-b968-06af27377c8e-tls-certs" pod "8c95874c-5d7b-4aeb-b968-06af27377c8e" (UID: "8c95874c-5d7b-4aeb-b968-06af27377c8e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:52:05.103480 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:05.103401 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c95874c-5d7b-4aeb-b968-06af27377c8e-kube-api-access-btdw9" pod "8c95874c-5d7b-4aeb-b968-06af27377c8e" (UID: "8c95874c-5d7b-4aeb-b968-06af27377c8e"). InnerVolumeSpecName "kube-api-access-btdw9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:52:05.202802 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:05.202725 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btdw9\" (UniqueName: \"kubernetes.io/projected/8c95874c-5d7b-4aeb-b968-06af27377c8e-kube-api-access-btdw9\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:52:05.202802 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:05.202755 6960 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c95874c-5d7b-4aeb-b968-06af27377c8e-dshm\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:52:05.202802 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:05.202766 6960 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c95874c-5d7b-4aeb-b968-06af27377c8e-tls-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:52:05.202802 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:05.202775 6960 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c95874c-5d7b-4aeb-b968-06af27377c8e-tmp-dir\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:52:05.242779 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:05.242751 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-45f649-dockercfg-jndg9\"" Oct 30 18:52:05.440734 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:05.440700 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:05.495414 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:52:05.495307 6960 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-ipp-b-45f649: failed to sync configmap cache: timed out waiting for the condition Oct 30 18:52:05.495578 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:52:05.495326 6960 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-ipp-b-45f649: failed to sync configmap cache: timed out waiting for the condition Oct 30 18:52:05.495776 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:52:05.495528 6960 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a7426960-9456-44cc-9de3-13158003510b-plugins-config-volume podName:a7426960-9456-44cc-9de3-13158003510b nodeName:}" failed. No retries permitted until 2026-10-30 18:52:05.995504694 +0000 UTC m=+1093.524535823 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/a7426960-9456-44cc-9de3-13158003510b-plugins-config-volume") pod "payload-processing-e2e-ipp-b-45f649-7564cc6b9d-rvfdp" (UID: "a7426960-9456-44cc-9de3-13158003510b") : failed to sync configmap cache: timed out waiting for the condition Oct 30 18:52:05.495858 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:52:05.495797 6960 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/09fe7720-3d9d-4327-a6a5-b94bbcff94ca-plugins-config-volume podName:09fe7720-3d9d-4327-a6a5-b94bbcff94ca nodeName:}" failed. No retries permitted until 2026-10-30 18:52:05.995778588 +0000 UTC m=+1093.524809719 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/09fe7720-3d9d-4327-a6a5-b94bbcff94ca-plugins-config-volume") pod "payload-pre-processing-e2e-ipp-b-45f649-849bdd96c9-mwhwd" (UID: "09fe7720-3d9d-4327-a6a5-b94bbcff94ca") : failed to sync configmap cache: timed out waiting for the condition Oct 30 18:52:05.569917 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:05.569881 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-45f649\"" Oct 30 18:52:05.614444 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:05.614412 6960 generic.go:356] "Generic (PLEG): container finished" podID="8c95874c-5d7b-4aeb-b968-06af27377c8e" containerID="9bb352a012907d22139064c0a28767116c4c0d317a5ca9ba5567d79abb6a6e0b" exitCode=0 Oct 30 18:52:05.614626 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:05.614458 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-2aada387/e2e-lifecycle-model-2aada387-kserve-dcfbdc957-48c2n" Oct 30 18:52:05.614626 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:05.614494 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-2aada387/e2e-lifecycle-model-2aada387-kserve-dcfbdc957-48c2n" event={"ID":"8c95874c-5d7b-4aeb-b968-06af27377c8e","Type":"ContainerDied","Data":"9bb352a012907d22139064c0a28767116c4c0d317a5ca9ba5567d79abb6a6e0b"} Oct 30 18:52:05.614626 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:05.614539 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-2aada387/e2e-lifecycle-model-2aada387-kserve-dcfbdc957-48c2n" event={"ID":"8c95874c-5d7b-4aeb-b968-06af27377c8e","Type":"ContainerDied","Data":"bf2325986500fd10141f10ae757dd7f7889e6eaab1978b666699c695be1bf6ef"} Oct 30 18:52:05.614626 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:05.614559 6960 scope.go:122] "RemoveContainer" containerID="9bb352a012907d22139064c0a28767116c4c0d317a5ca9ba5567d79abb6a6e0b" Oct 30 18:52:05.646272 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:05.646247 6960 scope.go:122] "RemoveContainer" containerID="9bb352a012907d22139064c0a28767116c4c0d317a5ca9ba5567d79abb6a6e0b" Oct 30 18:52:05.646956 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:52:05.646930 6960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bb352a012907d22139064c0a28767116c4c0d317a5ca9ba5567d79abb6a6e0b\": container with ID starting with 9bb352a012907d22139064c0a28767116c4c0d317a5ca9ba5567d79abb6a6e0b not found: ID does not exist" containerID="9bb352a012907d22139064c0a28767116c4c0d317a5ca9ba5567d79abb6a6e0b" Oct 30 18:52:05.647032 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:05.646973 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bb352a012907d22139064c0a28767116c4c0d317a5ca9ba5567d79abb6a6e0b"} err="failed to get container status \"9bb352a012907d22139064c0a28767116c4c0d317a5ca9ba5567d79abb6a6e0b\": rpc error: code = NotFound desc = could not find container \"9bb352a012907d22139064c0a28767116c4c0d317a5ca9ba5567d79abb6a6e0b\": container with ID starting with 9bb352a012907d22139064c0a28767116c4c0d317a5ca9ba5567d79abb6a6e0b not found: ID does not exist" Oct 30 18:52:05.665181 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:05.665104 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-2aada387/e2e-lifecycle-model-2aada387-kserve-dcfbdc957-48c2n"] Oct 30 18:52:05.676441 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:05.676374 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-2aada387/e2e-lifecycle-model-2aada387-kserve-dcfbdc957-48c2n"] Oct 30 18:52:05.699819 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:05.699785 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:05.839984 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:05.839947 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:06.015431 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:06.015397 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09fe7720-3d9d-4327-a6a5-b94bbcff94ca-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-45f649-849bdd96c9-mwhwd\" (UID: \"09fe7720-3d9d-4327-a6a5-b94bbcff94ca\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45f649-849bdd96c9-mwhwd" Oct 30 18:52:06.015575 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:06.015555 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7426960-9456-44cc-9de3-13158003510b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-45f649-7564cc6b9d-rvfdp\" (UID: \"a7426960-9456-44cc-9de3-13158003510b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-45f649-7564cc6b9d-rvfdp" Oct 30 18:52:06.016228 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:06.016210 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7426960-9456-44cc-9de3-13158003510b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-45f649-7564cc6b9d-rvfdp\" (UID: \"a7426960-9456-44cc-9de3-13158003510b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-45f649-7564cc6b9d-rvfdp" Oct 30 18:52:06.016282 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:06.016263 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09fe7720-3d9d-4327-a6a5-b94bbcff94ca-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-45f649-849bdd96c9-mwhwd\" (UID: \"09fe7720-3d9d-4327-a6a5-b94bbcff94ca\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45f649-849bdd96c9-mwhwd" Oct 30 18:52:06.210778 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:06.210698 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45f649-849bdd96c9-mwhwd" Oct 30 18:52:06.231489 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:06.231227 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-45f649-7564cc6b9d-rvfdp" Oct 30 18:52:06.510474 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:06.510393 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-45f649-849bdd96c9-mwhwd"] Oct 30 18:52:06.576356 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:06.576332 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-45f649-7564cc6b9d-rvfdp"] Oct 30 18:52:06.577948 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:52:06.577921 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7426960_9456_44cc_9de3_13158003510b.slice/crio-12ce05f5de3a326717ac101b673ccc42864bdb33a78409bea8ac751d7d7faf16 WatchSource:0}: Error finding container 12ce05f5de3a326717ac101b673ccc42864bdb33a78409bea8ac751d7d7faf16: Status 404 returned error can't find the container with id 12ce05f5de3a326717ac101b673ccc42864bdb33a78409bea8ac751d7d7faf16 Oct 30 18:52:06.597182 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:06.597159 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c95874c-5d7b-4aeb-b968-06af27377c8e" path="/var/lib/kubelet/pods/8c95874c-5d7b-4aeb-b968-06af27377c8e/volumes" Oct 30 18:52:06.623897 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:06.623865 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-45f649-7564cc6b9d-rvfdp" event={"ID":"a7426960-9456-44cc-9de3-13158003510b","Type":"ContainerStarted","Data":"12ce05f5de3a326717ac101b673ccc42864bdb33a78409bea8ac751d7d7faf16"} Oct 30 18:52:06.626847 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:06.626819 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45f649-849bdd96c9-mwhwd" event={"ID":"09fe7720-3d9d-4327-a6a5-b94bbcff94ca","Type":"ContainerStarted","Data":"6b2af5579a13781611fe25da0fd91307e9581f494546c49c26097835cf10397d"} Oct 30 18:52:06.912765 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:06.912731 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:07.024963 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:07.024920 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9ceefc-7767fff8c4-k4tpl"] Oct 30 18:52:07.026185 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:07.026116 6960 cpu_manager.go:414] "containerMap: removing container" podUID="8c95874c-5d7b-4aeb-b968-06af27377c8e" containerName="main" Oct 30 18:52:07.026185 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:07.026166 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c95874c-5d7b-4aeb-b968-06af27377c8e" containerName="main" Oct 30 18:52:07.026431 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:07.026414 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c95874c-5d7b-4aeb-b968-06af27377c8e" containerName="main" Oct 30 18:52:07.030450 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:07.030428 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-9ceefc-7767fff8c4-k4tpl" Oct 30 18:52:07.039286 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:07.039021 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b64e1bad-fa24-4c1d-84a4-512af66f8feb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-9ceefc-7767fff8c4-k4tpl\" (UID: \"b64e1bad-fa24-4c1d-84a4-512af66f8feb\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9ceefc-7767fff8c4-k4tpl" Oct 30 18:52:07.039286 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:07.039157 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bxx9\" (UniqueName: \"kubernetes.io/projected/b64e1bad-fa24-4c1d-84a4-512af66f8feb-kube-api-access-4bxx9\") pod \"payload-processing-e2e-ipp-a-9ceefc-7767fff8c4-k4tpl\" (UID: \"b64e1bad-fa24-4c1d-84a4-512af66f8feb\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9ceefc-7767fff8c4-k4tpl" Oct 30 18:52:07.057168 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:07.050788 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9ceefc-7767fff8c4-k4tpl"] Oct 30 18:52:07.057168 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:07.052280 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:07.110013 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:07.108640 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-9ceefc\"" Oct 30 18:52:07.129821 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:07.129750 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-9ceefc-dockercfg-ltm7z\"" Oct 30 18:52:07.155866 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:07.154339 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b64e1bad-fa24-4c1d-84a4-512af66f8feb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-9ceefc-7767fff8c4-k4tpl\" (UID: \"b64e1bad-fa24-4c1d-84a4-512af66f8feb\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9ceefc-7767fff8c4-k4tpl" Oct 30 18:52:07.155866 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:07.154502 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bxx9\" (UniqueName: \"kubernetes.io/projected/b64e1bad-fa24-4c1d-84a4-512af66f8feb-kube-api-access-4bxx9\") pod \"payload-processing-e2e-ipp-a-9ceefc-7767fff8c4-k4tpl\" (UID: \"b64e1bad-fa24-4c1d-84a4-512af66f8feb\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9ceefc-7767fff8c4-k4tpl" Oct 30 18:52:07.155866 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:07.155802 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b64e1bad-fa24-4c1d-84a4-512af66f8feb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-9ceefc-7767fff8c4-k4tpl\" (UID: \"b64e1bad-fa24-4c1d-84a4-512af66f8feb\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9ceefc-7767fff8c4-k4tpl" Oct 30 18:52:07.210274 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:07.210094 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bxx9\" (UniqueName: \"kubernetes.io/projected/b64e1bad-fa24-4c1d-84a4-512af66f8feb-kube-api-access-4bxx9\") pod \"payload-processing-e2e-ipp-a-9ceefc-7767fff8c4-k4tpl\" (UID: \"b64e1bad-fa24-4c1d-84a4-512af66f8feb\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9ceefc-7767fff8c4-k4tpl" Oct 30 18:52:07.360162 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:07.360114 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-9ceefc-7767fff8c4-k4tpl" Oct 30 18:52:07.634732 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:07.634695 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45f649-849bdd96c9-mwhwd" event={"ID":"09fe7720-3d9d-4327-a6a5-b94bbcff94ca","Type":"ContainerStarted","Data":"f5f4771a95a3f0e27927fae16129ca714d7b603894357e985cd07dfe493b371a"} Oct 30 18:52:07.634908 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:07.634758 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45f649-849bdd96c9-mwhwd" Oct 30 18:52:07.636453 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:07.636422 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-45f649-7564cc6b9d-rvfdp" event={"ID":"a7426960-9456-44cc-9de3-13158003510b","Type":"ContainerStarted","Data":"07a52f7f44169651061f271ae341d3e850797a52ea9aacbcbfe1d36756a06f52"} Oct 30 18:52:07.636592 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:07.636484 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-45f649-7564cc6b9d-rvfdp" Oct 30 18:52:07.641531 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:52:07.641506 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb64e1bad_fa24_4c1d_84a4_512af66f8feb.slice/crio-8ed2cef4cbe25440b236dae5db5a91e6fe6ea2cb2415a66391fa03537df86b37 WatchSource:0}: Error finding container 8ed2cef4cbe25440b236dae5db5a91e6fe6ea2cb2415a66391fa03537df86b37: Status 404 returned error can't find the container with id 8ed2cef4cbe25440b236dae5db5a91e6fe6ea2cb2415a66391fa03537df86b37 Oct 30 18:52:07.641681 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:07.641665 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9ceefc-7767fff8c4-k4tpl"] Oct 30 18:52:07.663240 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:07.663187 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45f649-849bdd96c9-mwhwd" podStartSLOduration=3.663169104 podStartE2EDuration="3.663169104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:52:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:07.657958346 +0000 UTC m=+1095.186989484" watchObservedRunningTime="2026-10-30 18:52:07.663169104 +0000 UTC m=+1095.192200243" Oct 30 18:52:07.679273 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:07.679221 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-45f649-7564cc6b9d-rvfdp" podStartSLOduration=3.6791386790000002 podStartE2EDuration="3.679138679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:52:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:07.676079908 +0000 UTC m=+1095.205111046" watchObservedRunningTime="2026-10-30 18:52:07.679138679 +0000 UTC m=+1095.208169840" Oct 30 18:52:07.870352 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:07.865675 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8fd4ed-57674cb595-sdrlj"] Oct 30 18:52:07.873501 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:07.870774 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8fd4ed-57674cb595-sdrlj" Oct 30 18:52:07.885723 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:07.883271 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-8fd4ed-serving-cert\"" Oct 30 18:52:07.888058 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:07.888032 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8fd4ed-57674cb595-sdrlj"] Oct 30 18:52:07.970448 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:07.970416 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/358545ad-9045-403a-9d72-aa4297431ba1-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-8fd4ed-57674cb595-sdrlj\" (UID: \"358545ad-9045-403a-9d72-aa4297431ba1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8fd4ed-57674cb595-sdrlj" Oct 30 18:52:07.970627 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:07.970467 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/358545ad-9045-403a-9d72-aa4297431ba1-maas-api-tls\") pod \"maas-api-e2e-ipp-b-8fd4ed-57674cb595-sdrlj\" (UID: \"358545ad-9045-403a-9d72-aa4297431ba1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8fd4ed-57674cb595-sdrlj" Oct 30 18:52:07.970627 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:07.970539 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w5dg\" (UniqueName: \"kubernetes.io/projected/358545ad-9045-403a-9d72-aa4297431ba1-kube-api-access-7w5dg\") pod \"maas-api-e2e-ipp-b-8fd4ed-57674cb595-sdrlj\" (UID: \"358545ad-9045-403a-9d72-aa4297431ba1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8fd4ed-57674cb595-sdrlj" Oct 30 18:52:07.970627 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:07.970612 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/358545ad-9045-403a-9d72-aa4297431ba1-metrics-certs\") pod \"maas-api-e2e-ipp-b-8fd4ed-57674cb595-sdrlj\" (UID: \"358545ad-9045-403a-9d72-aa4297431ba1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8fd4ed-57674cb595-sdrlj" Oct 30 18:52:07.996044 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:07.996014 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:08.072314 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:08.072283 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/358545ad-9045-403a-9d72-aa4297431ba1-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-8fd4ed-57674cb595-sdrlj\" (UID: \"358545ad-9045-403a-9d72-aa4297431ba1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8fd4ed-57674cb595-sdrlj" Oct 30 18:52:08.072498 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:08.072332 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/358545ad-9045-403a-9d72-aa4297431ba1-maas-api-tls\") pod \"maas-api-e2e-ipp-b-8fd4ed-57674cb595-sdrlj\" (UID: \"358545ad-9045-403a-9d72-aa4297431ba1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8fd4ed-57674cb595-sdrlj" Oct 30 18:52:08.072498 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:08.072379 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w5dg\" (UniqueName: \"kubernetes.io/projected/358545ad-9045-403a-9d72-aa4297431ba1-kube-api-access-7w5dg\") pod \"maas-api-e2e-ipp-b-8fd4ed-57674cb595-sdrlj\" (UID: \"358545ad-9045-403a-9d72-aa4297431ba1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8fd4ed-57674cb595-sdrlj" Oct 30 18:52:08.072498 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:08.072408 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/358545ad-9045-403a-9d72-aa4297431ba1-metrics-certs\") pod \"maas-api-e2e-ipp-b-8fd4ed-57674cb595-sdrlj\" (UID: \"358545ad-9045-403a-9d72-aa4297431ba1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8fd4ed-57674cb595-sdrlj" Oct 30 18:52:08.072968 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:08.072946 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/358545ad-9045-403a-9d72-aa4297431ba1-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-8fd4ed-57674cb595-sdrlj\" (UID: \"358545ad-9045-403a-9d72-aa4297431ba1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8fd4ed-57674cb595-sdrlj" Oct 30 18:52:08.075250 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:08.075224 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/358545ad-9045-403a-9d72-aa4297431ba1-metrics-certs\") pod \"maas-api-e2e-ipp-b-8fd4ed-57674cb595-sdrlj\" (UID: \"358545ad-9045-403a-9d72-aa4297431ba1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8fd4ed-57674cb595-sdrlj" Oct 30 18:52:08.075781 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:08.075751 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/358545ad-9045-403a-9d72-aa4297431ba1-maas-api-tls\") pod \"maas-api-e2e-ipp-b-8fd4ed-57674cb595-sdrlj\" (UID: \"358545ad-9045-403a-9d72-aa4297431ba1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8fd4ed-57674cb595-sdrlj" Oct 30 18:52:08.084910 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:08.084882 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w5dg\" (UniqueName: \"kubernetes.io/projected/358545ad-9045-403a-9d72-aa4297431ba1-kube-api-access-7w5dg\") pod \"maas-api-e2e-ipp-b-8fd4ed-57674cb595-sdrlj\" (UID: \"358545ad-9045-403a-9d72-aa4297431ba1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8fd4ed-57674cb595-sdrlj" Oct 30 18:52:08.112289 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:08.112252 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:08.192762 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:08.192677 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8fd4ed-57674cb595-sdrlj" Oct 30 18:52:08.492758 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:52:08.492713 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod358545ad_9045_403a_9d72_aa4297431ba1.slice/crio-3eb2dbb73519a640501cc0ba739925981e44eb597f4a071e3b9598ee09b37990 WatchSource:0}: Error finding container 3eb2dbb73519a640501cc0ba739925981e44eb597f4a071e3b9598ee09b37990: Status 404 returned error can't find the container with id 3eb2dbb73519a640501cc0ba739925981e44eb597f4a071e3b9598ee09b37990 Oct 30 18:52:08.493940 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:08.493914 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8fd4ed-57674cb595-sdrlj"] Oct 30 18:52:08.645926 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:08.645877 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8fd4ed-57674cb595-sdrlj" event={"ID":"358545ad-9045-403a-9d72-aa4297431ba1","Type":"ContainerStarted","Data":"3eb2dbb73519a640501cc0ba739925981e44eb597f4a071e3b9598ee09b37990"} Oct 30 18:52:08.648201 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:08.648160 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-9ceefc-7767fff8c4-k4tpl" event={"ID":"b64e1bad-fa24-4c1d-84a4-512af66f8feb","Type":"ContainerStarted","Data":"b8342f9c2466386c07a1ec4b7c57f40ed9d140074b9b47d84670855350be46d2"} Oct 30 18:52:08.648398 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:08.648376 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-9ceefc-7767fff8c4-k4tpl" event={"ID":"b64e1bad-fa24-4c1d-84a4-512af66f8feb","Type":"ContainerStarted","Data":"8ed2cef4cbe25440b236dae5db5a91e6fe6ea2cb2415a66391fa03537df86b37"} Oct 30 18:52:08.671720 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:08.671675 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-9ceefc-7767fff8c4-k4tpl" podStartSLOduration=2.671662704 podStartE2EDuration="2.671662704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:52:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:08.667000544 +0000 UTC m=+1096.196031679" watchObservedRunningTime="2026-10-30 18:52:08.671662704 +0000 UTC m=+1096.200693842" Oct 30 18:52:08.917736 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:08.917707 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:09.656552 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:09.656507 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8fd4ed-57674cb595-sdrlj" event={"ID":"358545ad-9045-403a-9d72-aa4297431ba1","Type":"ContainerStarted","Data":"85ff683edaaa9f1c13e994d029465ab53188b403f7e84e846bbb91d7ea22f6fd"} Oct 30 18:52:09.657063 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:09.656787 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8fd4ed-57674cb595-sdrlj" Oct 30 18:52:09.657063 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:09.656831 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-9ceefc-7767fff8c4-k4tpl" Oct 30 18:52:09.727466 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:09.727418 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8fd4ed-57674cb595-sdrlj" podStartSLOduration=2.310473321 podStartE2EDuration="2.727404227s" totalImagesPullingTime="416.930906ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:52:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:09.724666586 +0000 UTC m=+1097.253697727" watchObservedRunningTime="2026-10-30 18:52:09.727404227 +0000 UTC m=+1097.256435366" Oct 30 18:52:10.100884 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:10.100850 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:10.917962 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:10.917923 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:11.030349 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:11.030314 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:11.179184 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:11.179064 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:11.941103 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:11.941055 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:12.057491 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:12.057452 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:12.919185 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:12.919124 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:13.894307 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:13.894271 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:14.822431 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:14.819687 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-45f649-7577cd8b64-w75hv"] Oct 30 18:52:14.824681 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:14.824655 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45f649-7577cd8b64-w75hv" Oct 30 18:52:14.833715 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:14.833691 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-45f649-serving-cert\"" Oct 30 18:52:14.847940 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:14.847915 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-45f649-7577cd8b64-w75hv"] Oct 30 18:52:14.886790 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:14.886755 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/364b132c-231b-4108-9808-7ebbef4a9fbc-metrics-certs\") pod \"maas-api-e2e-ipp-b-45f649-7577cd8b64-w75hv\" (UID: \"364b132c-231b-4108-9808-7ebbef4a9fbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45f649-7577cd8b64-w75hv" Oct 30 18:52:14.886790 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:14.886798 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/364b132c-231b-4108-9808-7ebbef4a9fbc-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-45f649-7577cd8b64-w75hv\" (UID: \"364b132c-231b-4108-9808-7ebbef4a9fbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45f649-7577cd8b64-w75hv" Oct 30 18:52:14.887038 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:14.887014 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/364b132c-231b-4108-9808-7ebbef4a9fbc-maas-api-tls\") pod \"maas-api-e2e-ipp-b-45f649-7577cd8b64-w75hv\" (UID: \"364b132c-231b-4108-9808-7ebbef4a9fbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45f649-7577cd8b64-w75hv" Oct 30 18:52:14.887112 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:14.887070 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjh4v\" (UniqueName: \"kubernetes.io/projected/364b132c-231b-4108-9808-7ebbef4a9fbc-kube-api-access-kjh4v\") pod \"maas-api-e2e-ipp-b-45f649-7577cd8b64-w75hv\" (UID: \"364b132c-231b-4108-9808-7ebbef4a9fbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45f649-7577cd8b64-w75hv" Oct 30 18:52:14.887288 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:14.887267 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:14.997472 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:14.997436 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:14.999216 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:14.999176 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/364b132c-231b-4108-9808-7ebbef4a9fbc-metrics-certs\") pod \"maas-api-e2e-ipp-b-45f649-7577cd8b64-w75hv\" (UID: \"364b132c-231b-4108-9808-7ebbef4a9fbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45f649-7577cd8b64-w75hv" Oct 30 18:52:14.999917 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:14.999897 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/364b132c-231b-4108-9808-7ebbef4a9fbc-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-45f649-7577cd8b64-w75hv\" (UID: \"364b132c-231b-4108-9808-7ebbef4a9fbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45f649-7577cd8b64-w75hv" Oct 30 18:52:15.001182 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:15.000887 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/364b132c-231b-4108-9808-7ebbef4a9fbc-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-45f649-7577cd8b64-w75hv\" (UID: \"364b132c-231b-4108-9808-7ebbef4a9fbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45f649-7577cd8b64-w75hv" Oct 30 18:52:15.001386 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:15.001367 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/364b132c-231b-4108-9808-7ebbef4a9fbc-maas-api-tls\") pod \"maas-api-e2e-ipp-b-45f649-7577cd8b64-w75hv\" (UID: \"364b132c-231b-4108-9808-7ebbef4a9fbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45f649-7577cd8b64-w75hv" Oct 30 18:52:15.001999 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:15.001982 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjh4v\" (UniqueName: \"kubernetes.io/projected/364b132c-231b-4108-9808-7ebbef4a9fbc-kube-api-access-kjh4v\") pod \"maas-api-e2e-ipp-b-45f649-7577cd8b64-w75hv\" (UID: \"364b132c-231b-4108-9808-7ebbef4a9fbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45f649-7577cd8b64-w75hv" Oct 30 18:52:15.003037 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:15.002900 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/364b132c-231b-4108-9808-7ebbef4a9fbc-metrics-certs\") pod \"maas-api-e2e-ipp-b-45f649-7577cd8b64-w75hv\" (UID: \"364b132c-231b-4108-9808-7ebbef4a9fbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45f649-7577cd8b64-w75hv" Oct 30 18:52:15.020194 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:15.018388 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjh4v\" (UniqueName: \"kubernetes.io/projected/364b132c-231b-4108-9808-7ebbef4a9fbc-kube-api-access-kjh4v\") pod \"maas-api-e2e-ipp-b-45f649-7577cd8b64-w75hv\" (UID: \"364b132c-231b-4108-9808-7ebbef4a9fbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45f649-7577cd8b64-w75hv" Oct 30 18:52:15.020554 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:15.020528 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/364b132c-231b-4108-9808-7ebbef4a9fbc-maas-api-tls\") pod \"maas-api-e2e-ipp-b-45f649-7577cd8b64-w75hv\" (UID: \"364b132c-231b-4108-9808-7ebbef4a9fbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45f649-7577cd8b64-w75hv" Oct 30 18:52:15.148280 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:15.148201 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45f649-7577cd8b64-w75hv" Oct 30 18:52:15.455221 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:15.452789 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-45f649-7577cd8b64-w75hv"] Oct 30 18:52:15.669780 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:15.669754 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8fd4ed-57674cb595-sdrlj" Oct 30 18:52:15.726169 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:15.726070 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45f649-7577cd8b64-w75hv" event={"ID":"364b132c-231b-4108-9808-7ebbef4a9fbc","Type":"ContainerStarted","Data":"d8df0a7b2acb26eb7a31c5bec7768e4a5c0212f309573992a74f7574c035e767"} Oct 30 18:52:15.817514 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:15.817480 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:16.734275 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:16.734237 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45f649-7577cd8b64-w75hv" event={"ID":"364b132c-231b-4108-9808-7ebbef4a9fbc","Type":"ContainerStarted","Data":"0da851fccf82b4be86a367340005bdfda38d52b2b42ff7c49dffd6e10a42420b"} Oct 30 18:52:16.734923 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:16.734343 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45f649-7577cd8b64-w75hv" Oct 30 18:52:16.757961 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:16.757923 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45f649-7577cd8b64-w75hv" podStartSLOduration=2.349401258 podStartE2EDuration="2.757908191s" totalImagesPullingTime="408.506933ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:52:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:16.757908634 +0000 UTC m=+1104.286939774" watchObservedRunningTime="2026-10-30 18:52:16.757908191 +0000 UTC m=+1104.286939329" Oct 30 18:52:17.010581 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:17.010500 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:17.873972 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:17.873943 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:17.978558 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:17.978513 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:18.090336 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:18.090299 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:18.651267 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:18.651242 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45f649-849bdd96c9-mwhwd" Oct 30 18:52:18.651434 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:18.651304 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-45f649-7564cc6b9d-rvfdp" Oct 30 18:52:18.847130 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:18.847007 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:20.102454 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:20.102393 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:20.666806 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:20.666780 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-9ceefc-7767fff8c4-k4tpl" Oct 30 18:52:20.912249 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:20.912205 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:21.739204 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:21.735120 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:22.698475 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:22.698447 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:22.746055 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:22.746034 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45f649-7577cd8b64-w75hv" Oct 30 18:52:22.905567 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:22.905529 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:23.049222 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:23.048978 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:23.629970 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:23.629919 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:23.933555 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:52:23.933476 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 18:52:24.576054 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:24.576021 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:25.461644 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:25.461606 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:26.328861 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:26.328825 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:26.617002 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:26.616905 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:27.610599 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:27.610548 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:27.728560 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:27.728533 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:28.639278 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:28.639241 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:30.061252 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:30.061212 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:30.176736 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:30.176697 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:30.297889 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:30.297854 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:31.035101 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:31.035070 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:32.054756 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:32.054713 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:33.144730 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:33.144697 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-45f649-7564cc6b9d-rvfdp"] Oct 30 18:52:33.164303 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:33.164275 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-45f649-849bdd96c9-mwhwd"] Oct 30 18:52:33.217051 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:33.217016 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:33.328913 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:33.328879 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:33.511103 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:33.510997 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:34.233703 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:34.233669 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:34.675337 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:34.675306 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:34.809821 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:34.809779 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:34.986828 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:34.986729 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:35.137792 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:35.137747 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:35.618017 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:35.617984 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:35.756198 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:35.755422 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:35.886529 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:35.886452 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:36.208967 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:36.208895 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9ceefc-7767fff8c4-k4tpl"] Oct 30 18:52:36.340355 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:36.340313 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:36.467260 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:36.467175 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:36.600036 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:36.599992 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:36.824879 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:36.824845 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:36.991174 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:36.985395 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:37.101994 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:37.101907 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:38.100304 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:38.100267 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:38.145912 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:38.145875 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-45f649-7564cc6b9d-rvfdp" podUID="a7426960-9456-44cc-9de3-13158003510b" containerName="payload-processing" containerID="cri-o://07a52f7f44169651061f271ae341d3e850797a52ea9aacbcbfe1d36756a06f52" gracePeriod=25 Oct 30 18:52:38.165530 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:38.165500 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45f649-849bdd96c9-mwhwd" podUID="09fe7720-3d9d-4327-a6a5-b94bbcff94ca" containerName="payload-pre-processing" containerID="cri-o://f5f4771a95a3f0e27927fae16129ca714d7b603894357e985cd07dfe493b371a" gracePeriod=25 Oct 30 18:52:38.456343 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:38.456317 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-45f649-7564cc6b9d-rvfdp" Oct 30 18:52:38.511190 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:38.511159 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45f649-849bdd96c9-mwhwd" Oct 30 18:52:38.560251 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:38.560223 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7426960-9456-44cc-9de3-13158003510b-kube-api-access-8mk4w\" (UniqueName: \"kubernetes.io/projected/a7426960-9456-44cc-9de3-13158003510b-kube-api-access-8mk4w\") pod \"a7426960-9456-44cc-9de3-13158003510b\" (UID: \"a7426960-9456-44cc-9de3-13158003510b\") " Oct 30 18:52:38.560453 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:38.560434 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7426960-9456-44cc-9de3-13158003510b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7426960-9456-44cc-9de3-13158003510b-plugins-config-volume\") pod \"a7426960-9456-44cc-9de3-13158003510b\" (UID: \"a7426960-9456-44cc-9de3-13158003510b\") " Oct 30 18:52:38.560542 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:38.560528 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09fe7720-3d9d-4327-a6a5-b94bbcff94ca-kube-api-access-8x56g\" (UniqueName: \"kubernetes.io/projected/09fe7720-3d9d-4327-a6a5-b94bbcff94ca-kube-api-access-8x56g\") pod \"09fe7720-3d9d-4327-a6a5-b94bbcff94ca\" (UID: \"09fe7720-3d9d-4327-a6a5-b94bbcff94ca\") " Oct 30 18:52:38.560601 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:38.560570 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09fe7720-3d9d-4327-a6a5-b94bbcff94ca-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09fe7720-3d9d-4327-a6a5-b94bbcff94ca-plugins-config-volume\") pod \"09fe7720-3d9d-4327-a6a5-b94bbcff94ca\" (UID: \"09fe7720-3d9d-4327-a6a5-b94bbcff94ca\") " Oct 30 18:52:38.560979 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:38.560951 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7426960-9456-44cc-9de3-13158003510b-plugins-config-volume" pod "a7426960-9456-44cc-9de3-13158003510b" (UID: "a7426960-9456-44cc-9de3-13158003510b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:52:38.561523 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:38.561493 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09fe7720-3d9d-4327-a6a5-b94bbcff94ca-plugins-config-volume" pod "09fe7720-3d9d-4327-a6a5-b94bbcff94ca" (UID: "09fe7720-3d9d-4327-a6a5-b94bbcff94ca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:52:38.568499 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:38.568467 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7426960-9456-44cc-9de3-13158003510b-kube-api-access-8mk4w" pod "a7426960-9456-44cc-9de3-13158003510b" (UID: "a7426960-9456-44cc-9de3-13158003510b"). InnerVolumeSpecName "kube-api-access-8mk4w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:52:38.570135 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:38.570103 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09fe7720-3d9d-4327-a6a5-b94bbcff94ca-kube-api-access-8x56g" pod "09fe7720-3d9d-4327-a6a5-b94bbcff94ca" (UID: "09fe7720-3d9d-4327-a6a5-b94bbcff94ca"). InnerVolumeSpecName "kube-api-access-8x56g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:52:38.662538 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:38.662461 6960 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7426960-9456-44cc-9de3-13158003510b-plugins-config-volume\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:52:38.662538 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:38.662487 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8x56g\" (UniqueName: \"kubernetes.io/projected/09fe7720-3d9d-4327-a6a5-b94bbcff94ca-kube-api-access-8x56g\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:52:38.662538 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:38.662497 6960 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09fe7720-3d9d-4327-a6a5-b94bbcff94ca-plugins-config-volume\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:52:38.662538 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:38.662506 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mk4w\" (UniqueName: \"kubernetes.io/projected/a7426960-9456-44cc-9de3-13158003510b-kube-api-access-8mk4w\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:52:38.724569 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:38.724529 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:38.831958 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:38.831921 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:38.896813 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:38.896783 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ecef08-7bcb64d759-bqlt4"] Oct 30 18:52:38.897554 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:38.897537 6960 cpu_manager.go:414] "containerMap: removing container" podUID="a7426960-9456-44cc-9de3-13158003510b" containerName="payload-processing" Oct 30 18:52:38.897554 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:38.897554 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7426960-9456-44cc-9de3-13158003510b" containerName="payload-processing" Oct 30 18:52:38.897679 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:38.897602 6960 cpu_manager.go:414] "containerMap: removing container" podUID="09fe7720-3d9d-4327-a6a5-b94bbcff94ca" containerName="payload-pre-processing" Oct 30 18:52:38.897679 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:38.897608 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09fe7720-3d9d-4327-a6a5-b94bbcff94ca" containerName="payload-pre-processing" Oct 30 18:52:38.897751 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:38.897705 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="09fe7720-3d9d-4327-a6a5-b94bbcff94ca" containerName="payload-pre-processing" Oct 30 18:52:38.897751 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:38.897721 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7426960-9456-44cc-9de3-13158003510b" containerName="payload-processing" Oct 30 18:52:38.901517 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:38.901502 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ecef08-7bcb64d759-bqlt4" Oct 30 18:52:38.912794 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:38.912725 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-ecef08\"" Oct 30 18:52:38.912794 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:38.912740 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-ecef08-dockercfg-87gbb\"" Oct 30 18:52:38.913720 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:38.913697 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ecef08-7bcb64d759-bqlt4"] Oct 30 18:52:38.925904 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:38.925876 6960 generic.go:356] "Generic (PLEG): container finished" podID="a7426960-9456-44cc-9de3-13158003510b" containerID="07a52f7f44169651061f271ae341d3e850797a52ea9aacbcbfe1d36756a06f52" exitCode=0 Oct 30 18:52:38.926052 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:38.925931 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-45f649-7564cc6b9d-rvfdp" event={"ID":"a7426960-9456-44cc-9de3-13158003510b","Type":"ContainerDied","Data":"07a52f7f44169651061f271ae341d3e850797a52ea9aacbcbfe1d36756a06f52"} Oct 30 18:52:38.926052 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:38.926016 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-45f649-7564cc6b9d-rvfdp" event={"ID":"a7426960-9456-44cc-9de3-13158003510b","Type":"ContainerDied","Data":"12ce05f5de3a326717ac101b673ccc42864bdb33a78409bea8ac751d7d7faf16"} Oct 30 18:52:38.926217 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:38.926053 6960 scope.go:122] "RemoveContainer" containerID="07a52f7f44169651061f271ae341d3e850797a52ea9aacbcbfe1d36756a06f52" Oct 30 18:52:38.926292 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:38.926275 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-45f649-7564cc6b9d-rvfdp" Oct 30 18:52:38.930442 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:38.930413 6960 generic.go:356] "Generic (PLEG): container finished" podID="09fe7720-3d9d-4327-a6a5-b94bbcff94ca" containerID="f5f4771a95a3f0e27927fae16129ca714d7b603894357e985cd07dfe493b371a" exitCode=0 Oct 30 18:52:38.930575 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:38.930446 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45f649-849bdd96c9-mwhwd" Oct 30 18:52:38.930575 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:38.930500 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45f649-849bdd96c9-mwhwd" event={"ID":"09fe7720-3d9d-4327-a6a5-b94bbcff94ca","Type":"ContainerDied","Data":"f5f4771a95a3f0e27927fae16129ca714d7b603894357e985cd07dfe493b371a"} Oct 30 18:52:38.930575 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:38.930545 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-45f649-849bdd96c9-mwhwd" event={"ID":"09fe7720-3d9d-4327-a6a5-b94bbcff94ca","Type":"ContainerDied","Data":"6b2af5579a13781611fe25da0fd91307e9581f494546c49c26097835cf10397d"} Oct 30 18:52:38.942118 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:38.942100 6960 scope.go:122] "RemoveContainer" containerID="07a52f7f44169651061f271ae341d3e850797a52ea9aacbcbfe1d36756a06f52" Oct 30 18:52:38.942491 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:52:38.942460 6960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07a52f7f44169651061f271ae341d3e850797a52ea9aacbcbfe1d36756a06f52\": container with ID starting with 07a52f7f44169651061f271ae341d3e850797a52ea9aacbcbfe1d36756a06f52 not found: ID does not exist" containerID="07a52f7f44169651061f271ae341d3e850797a52ea9aacbcbfe1d36756a06f52" Oct 30 18:52:38.942612 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:38.942497 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07a52f7f44169651061f271ae341d3e850797a52ea9aacbcbfe1d36756a06f52"} err="failed to get container status \"07a52f7f44169651061f271ae341d3e850797a52ea9aacbcbfe1d36756a06f52\": rpc error: code = NotFound desc = could not find container \"07a52f7f44169651061f271ae341d3e850797a52ea9aacbcbfe1d36756a06f52\": container with ID starting with 07a52f7f44169651061f271ae341d3e850797a52ea9aacbcbfe1d36756a06f52 not found: ID does not exist" Oct 30 18:52:38.942612 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:38.942518 6960 scope.go:122] "RemoveContainer" containerID="f5f4771a95a3f0e27927fae16129ca714d7b603894357e985cd07dfe493b371a" Oct 30 18:52:38.955406 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:38.955382 6960 scope.go:122] "RemoveContainer" containerID="f5f4771a95a3f0e27927fae16129ca714d7b603894357e985cd07dfe493b371a" Oct 30 18:52:39.002216 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:38.995291 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-45f649-7564cc6b9d-rvfdp"] Oct 30 18:52:39.002216 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:38.997693 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-45f649-7564cc6b9d-rvfdp"] Oct 30 18:52:39.002216 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:52:38.999331 6960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5f4771a95a3f0e27927fae16129ca714d7b603894357e985cd07dfe493b371a\": container with ID starting with f5f4771a95a3f0e27927fae16129ca714d7b603894357e985cd07dfe493b371a not found: ID does not exist" containerID="f5f4771a95a3f0e27927fae16129ca714d7b603894357e985cd07dfe493b371a" Oct 30 18:52:39.002216 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:38.999368 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5f4771a95a3f0e27927fae16129ca714d7b603894357e985cd07dfe493b371a"} err="failed to get container status \"f5f4771a95a3f0e27927fae16129ca714d7b603894357e985cd07dfe493b371a\": rpc error: code = NotFound desc = could not find container \"f5f4771a95a3f0e27927fae16129ca714d7b603894357e985cd07dfe493b371a\": container with ID starting with f5f4771a95a3f0e27927fae16129ca714d7b603894357e985cd07dfe493b371a not found: ID does not exist" Oct 30 18:52:39.038190 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:39.031245 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hx46q\" (UniqueName: \"kubernetes.io/projected/4a18b92c-2dd3-4a63-8eb4-0fdb4f34e470-kube-api-access-hx46q\") pod \"payload-pre-processing-e2e-ipp-b-ecef08-7bcb64d759-bqlt4\" (UID: \"4a18b92c-2dd3-4a63-8eb4-0fdb4f34e470\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ecef08-7bcb64d759-bqlt4" Oct 30 18:52:39.038190 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:39.031605 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a18b92c-2dd3-4a63-8eb4-0fdb4f34e470-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ecef08-7bcb64d759-bqlt4\" (UID: \"4a18b92c-2dd3-4a63-8eb4-0fdb4f34e470\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ecef08-7bcb64d759-bqlt4" Oct 30 18:52:39.152171 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:39.142228 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-45f649-849bdd96c9-mwhwd"] Oct 30 18:52:39.152171 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:39.143493 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a18b92c-2dd3-4a63-8eb4-0fdb4f34e470-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ecef08-7bcb64d759-bqlt4\" (UID: \"4a18b92c-2dd3-4a63-8eb4-0fdb4f34e470\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ecef08-7bcb64d759-bqlt4" Oct 30 18:52:39.152171 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:39.143634 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hx46q\" (UniqueName: \"kubernetes.io/projected/4a18b92c-2dd3-4a63-8eb4-0fdb4f34e470-kube-api-access-hx46q\") pod \"payload-pre-processing-e2e-ipp-b-ecef08-7bcb64d759-bqlt4\" (UID: \"4a18b92c-2dd3-4a63-8eb4-0fdb4f34e470\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ecef08-7bcb64d759-bqlt4" Oct 30 18:52:39.152171 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:39.144976 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a18b92c-2dd3-4a63-8eb4-0fdb4f34e470-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ecef08-7bcb64d759-bqlt4\" (UID: \"4a18b92c-2dd3-4a63-8eb4-0fdb4f34e470\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ecef08-7bcb64d759-bqlt4" Oct 30 18:52:39.152171 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:39.150434 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-45f649-849bdd96c9-mwhwd"] Oct 30 18:52:39.161128 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:39.161087 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hx46q\" (UniqueName: \"kubernetes.io/projected/4a18b92c-2dd3-4a63-8eb4-0fdb4f34e470-kube-api-access-hx46q\") pod \"payload-pre-processing-e2e-ipp-b-ecef08-7bcb64d759-bqlt4\" (UID: \"4a18b92c-2dd3-4a63-8eb4-0fdb4f34e470\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ecef08-7bcb64d759-bqlt4" Oct 30 18:52:39.217657 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:39.217628 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ecef08-7bcb64d759-bqlt4" Oct 30 18:52:39.355884 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:39.355856 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:39.545445 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:39.545420 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ecef08-7bcb64d759-bqlt4"] Oct 30 18:52:39.546536 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:52:39.546510 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a18b92c_2dd3_4a63_8eb4_0fdb4f34e470.slice/crio-01a25dfa049d1588f69d06cf1887a2d392ebc459b2a4d3a549381e348059046f WatchSource:0}: Error finding container 01a25dfa049d1588f69d06cf1887a2d392ebc459b2a4d3a549381e348059046f: Status 404 returned error can't find the container with id 01a25dfa049d1588f69d06cf1887a2d392ebc459b2a4d3a549381e348059046f Oct 30 18:52:39.939026 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:39.938991 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ecef08-7bcb64d759-bqlt4" event={"ID":"4a18b92c-2dd3-4a63-8eb4-0fdb4f34e470","Type":"ContainerStarted","Data":"01a25dfa049d1588f69d06cf1887a2d392ebc459b2a4d3a549381e348059046f"} Oct 30 18:52:40.247598 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:40.247486 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:40.595727 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:40.595696 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09fe7720-3d9d-4327-a6a5-b94bbcff94ca" path="/var/lib/kubelet/pods/09fe7720-3d9d-4327-a6a5-b94bbcff94ca/volumes" Oct 30 18:52:40.596198 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:40.596173 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7426960-9456-44cc-9de3-13158003510b" path="/var/lib/kubelet/pods/a7426960-9456-44cc-9de3-13158003510b/volumes" Oct 30 18:52:41.110369 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:41.110336 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:41.215578 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:41.215533 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-9ceefc-7767fff8c4-k4tpl" podUID="b64e1bad-fa24-4c1d-84a4-512af66f8feb" containerName="payload-processing" containerID="cri-o://b8342f9c2466386c07a1ec4b7c57f40ed9d140074b9b47d84670855350be46d2" gracePeriod=25 Oct 30 18:52:41.546297 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:41.546268 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-9ceefc-7767fff8c4-k4tpl" Oct 30 18:52:41.580084 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:41.580060 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b64e1bad-fa24-4c1d-84a4-512af66f8feb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b64e1bad-fa24-4c1d-84a4-512af66f8feb-plugins-config-volume\") pod \"b64e1bad-fa24-4c1d-84a4-512af66f8feb\" (UID: \"b64e1bad-fa24-4c1d-84a4-512af66f8feb\") " Oct 30 18:52:41.580251 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:41.580161 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b64e1bad-fa24-4c1d-84a4-512af66f8feb-kube-api-access-4bxx9\" (UniqueName: \"kubernetes.io/projected/b64e1bad-fa24-4c1d-84a4-512af66f8feb-kube-api-access-4bxx9\") pod \"b64e1bad-fa24-4c1d-84a4-512af66f8feb\" (UID: \"b64e1bad-fa24-4c1d-84a4-512af66f8feb\") " Oct 30 18:52:41.580484 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:41.580456 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b64e1bad-fa24-4c1d-84a4-512af66f8feb-plugins-config-volume" pod "b64e1bad-fa24-4c1d-84a4-512af66f8feb" (UID: "b64e1bad-fa24-4c1d-84a4-512af66f8feb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:52:41.580852 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:41.580830 6960 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b64e1bad-fa24-4c1d-84a4-512af66f8feb-plugins-config-volume\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:52:41.582964 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:41.582937 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b64e1bad-fa24-4c1d-84a4-512af66f8feb-kube-api-access-4bxx9" pod "b64e1bad-fa24-4c1d-84a4-512af66f8feb" (UID: "b64e1bad-fa24-4c1d-84a4-512af66f8feb"). InnerVolumeSpecName "kube-api-access-4bxx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:52:41.683310 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:41.683280 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bxx9\" (UniqueName: \"kubernetes.io/projected/b64e1bad-fa24-4c1d-84a4-512af66f8feb-kube-api-access-4bxx9\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:52:41.961536 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:41.961434 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ecef08-7bcb64d759-bqlt4" event={"ID":"4a18b92c-2dd3-4a63-8eb4-0fdb4f34e470","Type":"ContainerStarted","Data":"8c4e66a797da4f97eb8f7420fc931535fff1a51814b872121efe18f089cd149e"} Oct 30 18:52:41.961536 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:41.961536 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ecef08-7bcb64d759-bqlt4" Oct 30 18:52:41.963579 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:41.963548 6960 generic.go:356] "Generic (PLEG): container finished" podID="b64e1bad-fa24-4c1d-84a4-512af66f8feb" containerID="b8342f9c2466386c07a1ec4b7c57f40ed9d140074b9b47d84670855350be46d2" exitCode=0 Oct 30 18:52:41.963713 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:41.963599 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-9ceefc-7767fff8c4-k4tpl" Oct 30 18:52:41.963713 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:41.963622 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-9ceefc-7767fff8c4-k4tpl" event={"ID":"b64e1bad-fa24-4c1d-84a4-512af66f8feb","Type":"ContainerDied","Data":"b8342f9c2466386c07a1ec4b7c57f40ed9d140074b9b47d84670855350be46d2"} Oct 30 18:52:41.963713 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:41.963669 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-9ceefc-7767fff8c4-k4tpl" event={"ID":"b64e1bad-fa24-4c1d-84a4-512af66f8feb","Type":"ContainerDied","Data":"8ed2cef4cbe25440b236dae5db5a91e6fe6ea2cb2415a66391fa03537df86b37"} Oct 30 18:52:41.963713 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:41.963694 6960 scope.go:122] "RemoveContainer" containerID="b8342f9c2466386c07a1ec4b7c57f40ed9d140074b9b47d84670855350be46d2" Oct 30 18:52:41.975554 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:41.975536 6960 scope.go:122] "RemoveContainer" containerID="b8342f9c2466386c07a1ec4b7c57f40ed9d140074b9b47d84670855350be46d2" Oct 30 18:52:41.975873 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:52:41.975854 6960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8342f9c2466386c07a1ec4b7c57f40ed9d140074b9b47d84670855350be46d2\": container with ID starting with b8342f9c2466386c07a1ec4b7c57f40ed9d140074b9b47d84670855350be46d2 not found: ID does not exist" containerID="b8342f9c2466386c07a1ec4b7c57f40ed9d140074b9b47d84670855350be46d2" Oct 30 18:52:41.975928 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:41.975883 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8342f9c2466386c07a1ec4b7c57f40ed9d140074b9b47d84670855350be46d2"} err="failed to get container status \"b8342f9c2466386c07a1ec4b7c57f40ed9d140074b9b47d84670855350be46d2\": rpc error: code = NotFound desc = could not find container \"b8342f9c2466386c07a1ec4b7c57f40ed9d140074b9b47d84670855350be46d2\": container with ID starting with b8342f9c2466386c07a1ec4b7c57f40ed9d140074b9b47d84670855350be46d2 not found: ID does not exist" Oct 30 18:52:41.982049 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:41.982002 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ecef08-7bcb64d759-bqlt4" podStartSLOduration=2.529709354 podStartE2EDuration="3.981986724s" totalImagesPullingTime="1.45227737s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:52:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:41.980477926 +0000 UTC m=+1129.509509064" watchObservedRunningTime="2026-10-30 18:52:41.981986724 +0000 UTC m=+1129.511017863" Oct 30 18:52:41.996373 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:41.996351 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9ceefc-7767fff8c4-k4tpl"] Oct 30 18:52:42.000994 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:42.000972 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9ceefc-7767fff8c4-k4tpl"] Oct 30 18:52:42.232317 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:42.232212 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:42.593111 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:42.593080 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b64e1bad-fa24-4c1d-84a4-512af66f8feb" path="/var/lib/kubelet/pods/b64e1bad-fa24-4c1d-84a4-512af66f8feb/volumes" Oct 30 18:52:42.864777 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:42.864686 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:42.974592 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:42.974559 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:43.114005 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:43.113970 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-45f649-6c66fcd59b-mv59d"] Oct 30 18:52:43.115103 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:43.115030 6960 cpu_manager.go:414] "containerMap: removing container" podUID="b64e1bad-fa24-4c1d-84a4-512af66f8feb" containerName="payload-processing" Oct 30 18:52:43.115103 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:43.115056 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b64e1bad-fa24-4c1d-84a4-512af66f8feb" containerName="payload-processing" Oct 30 18:52:43.115351 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:43.115246 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="b64e1bad-fa24-4c1d-84a4-512af66f8feb" containerName="payload-processing" Oct 30 18:52:43.159167 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:43.157216 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-45f649-6c66fcd59b-mv59d" Oct 30 18:52:43.221167 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:43.208444 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3a1f960-c8df-46f6-b3a7-9b5020863765-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-45f649-6c66fcd59b-mv59d\" (UID: \"d3a1f960-c8df-46f6-b3a7-9b5020863765\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-45f649-6c66fcd59b-mv59d" Oct 30 18:52:43.221167 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:43.208791 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7ldz\" (UniqueName: \"kubernetes.io/projected/d3a1f960-c8df-46f6-b3a7-9b5020863765-kube-api-access-s7ldz\") pod \"payload-processing-e2e-ipp-b-45f649-6c66fcd59b-mv59d\" (UID: \"d3a1f960-c8df-46f6-b3a7-9b5020863765\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-45f649-6c66fcd59b-mv59d" Oct 30 18:52:43.221167 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:43.212369 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-45f649-dockercfg-kntnk\"" Oct 30 18:52:43.221167 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:43.213830 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-45f649-6c66fcd59b-mv59d"] Oct 30 18:52:43.221167 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:43.216883 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-45f649\"" Oct 30 18:52:43.310509 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:43.310476 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7ldz\" (UniqueName: \"kubernetes.io/projected/d3a1f960-c8df-46f6-b3a7-9b5020863765-kube-api-access-s7ldz\") pod \"payload-processing-e2e-ipp-b-45f649-6c66fcd59b-mv59d\" (UID: \"d3a1f960-c8df-46f6-b3a7-9b5020863765\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-45f649-6c66fcd59b-mv59d" Oct 30 18:52:43.310936 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:43.310913 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3a1f960-c8df-46f6-b3a7-9b5020863765-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-45f649-6c66fcd59b-mv59d\" (UID: \"d3a1f960-c8df-46f6-b3a7-9b5020863765\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-45f649-6c66fcd59b-mv59d" Oct 30 18:52:43.312335 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:43.312021 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3a1f960-c8df-46f6-b3a7-9b5020863765-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-45f649-6c66fcd59b-mv59d\" (UID: \"d3a1f960-c8df-46f6-b3a7-9b5020863765\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-45f649-6c66fcd59b-mv59d" Oct 30 18:52:43.337724 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:43.337694 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7ldz\" (UniqueName: \"kubernetes.io/projected/d3a1f960-c8df-46f6-b3a7-9b5020863765-kube-api-access-s7ldz\") pod \"payload-processing-e2e-ipp-b-45f649-6c66fcd59b-mv59d\" (UID: \"d3a1f960-c8df-46f6-b3a7-9b5020863765\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-45f649-6c66fcd59b-mv59d" Oct 30 18:52:43.528559 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:43.528456 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-45f649-6c66fcd59b-mv59d" Oct 30 18:52:43.614778 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:43.614739 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:43.823934 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:52:43.823901 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3a1f960_c8df_46f6_b3a7_9b5020863765.slice/crio-5559b79a1f1af04a5094a8ed98723b6d8b0486f6ab3438812ef8cd1103c73aa3 WatchSource:0}: Error finding container 5559b79a1f1af04a5094a8ed98723b6d8b0486f6ab3438812ef8cd1103c73aa3: Status 404 returned error can't find the container with id 5559b79a1f1af04a5094a8ed98723b6d8b0486f6ab3438812ef8cd1103c73aa3 Oct 30 18:52:43.825260 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:43.824835 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-45f649-6c66fcd59b-mv59d"] Oct 30 18:52:43.981550 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:43.981501 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-45f649-6c66fcd59b-mv59d" event={"ID":"d3a1f960-c8df-46f6-b3a7-9b5020863765","Type":"ContainerStarted","Data":"655f00e7b6e35a7f3a28a0e14f39fb2afea0e2d03e9c2fb488847a584ca2a5e4"} Oct 30 18:52:43.981765 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:43.981577 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-45f649-6c66fcd59b-mv59d" event={"ID":"d3a1f960-c8df-46f6-b3a7-9b5020863765","Type":"ContainerStarted","Data":"5559b79a1f1af04a5094a8ed98723b6d8b0486f6ab3438812ef8cd1103c73aa3"} Oct 30 18:52:43.981765 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:43.981743 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-45f649-6c66fcd59b-mv59d" Oct 30 18:52:44.004290 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:44.004241 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-45f649-6c66fcd59b-mv59d" podStartSLOduration=1.004227035 podStartE2EDuration="1.004227035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:52:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:44.000767283 +0000 UTC m=+1131.529798446" watchObservedRunningTime="2026-10-30 18:52:44.004227035 +0000 UTC m=+1131.533258224" Oct 30 18:52:44.712351 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:44.712317 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:45.565766 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:45.565724 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:46.220772 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:46.220726 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:46.452814 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:46.452747 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:47.090954 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:47.090912 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:48.125640 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:48.125609 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:49.029873 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:49.029822 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:49.144243 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:49.144203 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:50.107855 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:50.107825 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:51.054163 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:51.054116 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:51.392096 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:51.392052 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:51.937319 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:51.937286 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:52.974097 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:52.974076 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ecef08-7bcb64d759-bqlt4" Oct 30 18:52:53.024526 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:53.024497 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:53.904818 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:53.904783 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:53.933732 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:52:53.933698 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 18:52:54.811032 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:54.810994 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:55.027320 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:55.027291 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-45f649-6c66fcd59b-mv59d" Oct 30 18:52:55.923057 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:55.923028 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:56.370819 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:56.370783 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:56.927385 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:56.927350 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:58.115527 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:58.115492 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:59.355218 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:52:59.355176 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:00.788171 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:00.787692 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:02.390027 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:02.389988 6960 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-dslrt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:53:02.390709 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:02.390669 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-dslrt" podUID="93b3f3a7-cd85-4996-88fe-7762155b3241" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:53:02.737414 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:02.737322 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:03.304428 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:03.304365 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:03.774428 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:03.774332 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:03.914744 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:03.914708 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:04.065549 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:04.065515 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:05.038174 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:05.038126 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:05.930235 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:05.930207 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:06.936801 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:06.936768 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:07.554419 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:07.554383 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:07.675921 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:07.675886 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:08.363935 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:08.363891 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:08.802793 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:08.802692 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:08.934649 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:08.934601 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:09.597929 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:09.597901 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:10.400261 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:10.400221 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:10.612758 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:10.612725 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:10.756309 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:10.756208 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:10.900479 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:10.900431 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:11.015809 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:11.015697 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:11.399565 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:11.399527 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:11.950757 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:11.950720 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:12.075295 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:12.075262 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:12.181552 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:12.181520 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:12.300672 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:12.300587 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:12.420671 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:12.420637 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:12.547622 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:12.547589 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:12.938873 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:12.938829 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:13.442991 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:13.442963 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:14.450708 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:14.450673 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5ad713-798d856f68-xqm68"] Oct 30 18:53:14.451355 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:14.451279 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5ad713-798d856f68-xqm68" podUID="ed0b8197-ddea-4c33-93c9-286d42ea7322" containerName="maas-api" containerID="cri-o://9c72f4f1e281421b0f7e969ec0a6fb9cca5d6fe4a34eb9573d673ad61e1d0b8e" gracePeriod=30 Oct 30 18:53:14.535368 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:14.535312 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:14.670166 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:14.669946 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:14.857197 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:14.841771 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:14.910166 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:14.907870 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5ad713-798d856f68-xqm68" Oct 30 18:53:14.999617 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:14.999570 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:15.072402 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:15.072364 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed0b8197-ddea-4c33-93c9-286d42ea7322-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed0b8197-ddea-4c33-93c9-286d42ea7322-metrics-certs\") pod \"ed0b8197-ddea-4c33-93c9-286d42ea7322\" (UID: \"ed0b8197-ddea-4c33-93c9-286d42ea7322\") " Oct 30 18:53:15.072615 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:15.072554 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed0b8197-ddea-4c33-93c9-286d42ea7322-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed0b8197-ddea-4c33-93c9-286d42ea7322-openshift-service-ca\") pod \"ed0b8197-ddea-4c33-93c9-286d42ea7322\" (UID: \"ed0b8197-ddea-4c33-93c9-286d42ea7322\") " Oct 30 18:53:15.072779 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:15.072754 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed0b8197-ddea-4c33-93c9-286d42ea7322-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed0b8197-ddea-4c33-93c9-286d42ea7322-maas-api-tls\") pod \"ed0b8197-ddea-4c33-93c9-286d42ea7322\" (UID: \"ed0b8197-ddea-4c33-93c9-286d42ea7322\") " Oct 30 18:53:15.072906 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:15.072885 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed0b8197-ddea-4c33-93c9-286d42ea7322-openshift-service-ca" pod "ed0b8197-ddea-4c33-93c9-286d42ea7322" (UID: "ed0b8197-ddea-4c33-93c9-286d42ea7322"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:53:15.073041 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:15.072927 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed0b8197-ddea-4c33-93c9-286d42ea7322-kube-api-access-tq7t4\" (UniqueName: \"kubernetes.io/projected/ed0b8197-ddea-4c33-93c9-286d42ea7322-kube-api-access-tq7t4\") pod \"ed0b8197-ddea-4c33-93c9-286d42ea7322\" (UID: \"ed0b8197-ddea-4c33-93c9-286d42ea7322\") " Oct 30 18:53:15.073825 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:15.073797 6960 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed0b8197-ddea-4c33-93c9-286d42ea7322-openshift-service-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:53:15.075212 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:15.075182 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed0b8197-ddea-4c33-93c9-286d42ea7322-metrics-certs" pod "ed0b8197-ddea-4c33-93c9-286d42ea7322" (UID: "ed0b8197-ddea-4c33-93c9-286d42ea7322"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:53:15.075470 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:15.075450 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed0b8197-ddea-4c33-93c9-286d42ea7322-maas-api-tls" pod "ed0b8197-ddea-4c33-93c9-286d42ea7322" (UID: "ed0b8197-ddea-4c33-93c9-286d42ea7322"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:53:15.075573 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:15.075551 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed0b8197-ddea-4c33-93c9-286d42ea7322-kube-api-access-tq7t4" pod "ed0b8197-ddea-4c33-93c9-286d42ea7322" (UID: "ed0b8197-ddea-4c33-93c9-286d42ea7322"). InnerVolumeSpecName "kube-api-access-tq7t4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:53:15.105413 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:15.105377 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:15.175601 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:15.175526 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tq7t4\" (UniqueName: \"kubernetes.io/projected/ed0b8197-ddea-4c33-93c9-286d42ea7322-kube-api-access-tq7t4\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:53:15.175601 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:15.175556 6960 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed0b8197-ddea-4c33-93c9-286d42ea7322-metrics-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:53:15.175601 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:15.175571 6960 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed0b8197-ddea-4c33-93c9-286d42ea7322-maas-api-tls\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:53:15.239489 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:15.239452 6960 generic.go:356] "Generic (PLEG): container finished" podID="ed0b8197-ddea-4c33-93c9-286d42ea7322" containerID="9c72f4f1e281421b0f7e969ec0a6fb9cca5d6fe4a34eb9573d673ad61e1d0b8e" exitCode=0 Oct 30 18:53:15.239676 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:15.239512 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5ad713-798d856f68-xqm68" event={"ID":"ed0b8197-ddea-4c33-93c9-286d42ea7322","Type":"ContainerDied","Data":"9c72f4f1e281421b0f7e969ec0a6fb9cca5d6fe4a34eb9573d673ad61e1d0b8e"} Oct 30 18:53:15.239676 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:15.239611 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5ad713-798d856f68-xqm68" event={"ID":"ed0b8197-ddea-4c33-93c9-286d42ea7322","Type":"ContainerDied","Data":"c263dc91a5f86fe33df3edcdaac3e593295fc81e25d868ce41668e204771f018"} Oct 30 18:53:15.239676 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:15.239669 6960 scope.go:122] "RemoveContainer" containerID="9c72f4f1e281421b0f7e969ec0a6fb9cca5d6fe4a34eb9573d673ad61e1d0b8e" Oct 30 18:53:15.239939 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:15.239918 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5ad713-798d856f68-xqm68" Oct 30 18:53:15.259509 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:15.259487 6960 scope.go:122] "RemoveContainer" containerID="9c72f4f1e281421b0f7e969ec0a6fb9cca5d6fe4a34eb9573d673ad61e1d0b8e" Oct 30 18:53:15.259800 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:53:15.259778 6960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c72f4f1e281421b0f7e969ec0a6fb9cca5d6fe4a34eb9573d673ad61e1d0b8e\": container with ID starting with 9c72f4f1e281421b0f7e969ec0a6fb9cca5d6fe4a34eb9573d673ad61e1d0b8e not found: ID does not exist" containerID="9c72f4f1e281421b0f7e969ec0a6fb9cca5d6fe4a34eb9573d673ad61e1d0b8e" Oct 30 18:53:15.259869 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:15.259809 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c72f4f1e281421b0f7e969ec0a6fb9cca5d6fe4a34eb9573d673ad61e1d0b8e"} err="failed to get container status \"9c72f4f1e281421b0f7e969ec0a6fb9cca5d6fe4a34eb9573d673ad61e1d0b8e\": rpc error: code = NotFound desc = could not find container \"9c72f4f1e281421b0f7e969ec0a6fb9cca5d6fe4a34eb9573d673ad61e1d0b8e\": container with ID starting with 9c72f4f1e281421b0f7e969ec0a6fb9cca5d6fe4a34eb9573d673ad61e1d0b8e not found: ID does not exist" Oct 30 18:53:15.269376 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:15.269355 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5ad713-798d856f68-xqm68"] Oct 30 18:53:15.277991 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:15.277968 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5ad713-798d856f68-xqm68"] Oct 30 18:53:15.521872 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:15.521786 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:15.908751 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:15.908717 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:16.594390 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:16.594359 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed0b8197-ddea-4c33-93c9-286d42ea7322" path="/var/lib/kubelet/pods/ed0b8197-ddea-4c33-93c9-286d42ea7322/volumes" Oct 30 18:53:16.916188 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:16.916088 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:17.815215 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:17.815175 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:18.577594 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:18.577558 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-348877-c6895b5c7-4m2kf"] Oct 30 18:53:18.578608 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:18.578585 6960 cpu_manager.go:414] "containerMap: removing container" podUID="ed0b8197-ddea-4c33-93c9-286d42ea7322" containerName="maas-api" Oct 30 18:53:18.578608 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:18.578605 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed0b8197-ddea-4c33-93c9-286d42ea7322" containerName="maas-api" Oct 30 18:53:18.578792 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:18.578780 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed0b8197-ddea-4c33-93c9-286d42ea7322" containerName="maas-api" Oct 30 18:53:18.584686 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:18.584660 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-348877-c6895b5c7-4m2kf" Oct 30 18:53:18.594653 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:18.594627 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-348877-serving-cert\"" Oct 30 18:53:18.601690 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:18.601662 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-348877-c6895b5c7-4m2kf"] Oct 30 18:53:18.671183 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:18.671132 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:18.733469 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:18.733427 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd7d5d70-3bf1-4b29-947a-0e7e175fabbb-maas-api-tls\") pod \"maas-api-e2e-ipp-b-348877-c6895b5c7-4m2kf\" (UID: \"bd7d5d70-3bf1-4b29-947a-0e7e175fabbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-348877-c6895b5c7-4m2kf" Oct 30 18:53:18.733678 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:18.733587 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd7d5d70-3bf1-4b29-947a-0e7e175fabbb-metrics-certs\") pod \"maas-api-e2e-ipp-b-348877-c6895b5c7-4m2kf\" (UID: \"bd7d5d70-3bf1-4b29-947a-0e7e175fabbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-348877-c6895b5c7-4m2kf" Oct 30 18:53:18.733678 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:18.733663 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdzkt\" (UniqueName: \"kubernetes.io/projected/bd7d5d70-3bf1-4b29-947a-0e7e175fabbb-kube-api-access-tdzkt\") pod \"maas-api-e2e-ipp-b-348877-c6895b5c7-4m2kf\" (UID: \"bd7d5d70-3bf1-4b29-947a-0e7e175fabbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-348877-c6895b5c7-4m2kf" Oct 30 18:53:18.735623 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:18.733798 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd7d5d70-3bf1-4b29-947a-0e7e175fabbb-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-348877-c6895b5c7-4m2kf\" (UID: \"bd7d5d70-3bf1-4b29-947a-0e7e175fabbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-348877-c6895b5c7-4m2kf" Oct 30 18:53:18.816516 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:18.816485 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:18.838354 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:18.838263 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd7d5d70-3bf1-4b29-947a-0e7e175fabbb-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-348877-c6895b5c7-4m2kf\" (UID: \"bd7d5d70-3bf1-4b29-947a-0e7e175fabbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-348877-c6895b5c7-4m2kf" Oct 30 18:53:18.838529 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:18.838367 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd7d5d70-3bf1-4b29-947a-0e7e175fabbb-maas-api-tls\") pod \"maas-api-e2e-ipp-b-348877-c6895b5c7-4m2kf\" (UID: \"bd7d5d70-3bf1-4b29-947a-0e7e175fabbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-348877-c6895b5c7-4m2kf" Oct 30 18:53:18.838529 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:18.838460 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd7d5d70-3bf1-4b29-947a-0e7e175fabbb-metrics-certs\") pod \"maas-api-e2e-ipp-b-348877-c6895b5c7-4m2kf\" (UID: \"bd7d5d70-3bf1-4b29-947a-0e7e175fabbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-348877-c6895b5c7-4m2kf" Oct 30 18:53:18.838529 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:18.838521 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdzkt\" (UniqueName: \"kubernetes.io/projected/bd7d5d70-3bf1-4b29-947a-0e7e175fabbb-kube-api-access-tdzkt\") pod \"maas-api-e2e-ipp-b-348877-c6895b5c7-4m2kf\" (UID: \"bd7d5d70-3bf1-4b29-947a-0e7e175fabbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-348877-c6895b5c7-4m2kf" Oct 30 18:53:18.838984 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:18.838963 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd7d5d70-3bf1-4b29-947a-0e7e175fabbb-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-348877-c6895b5c7-4m2kf\" (UID: \"bd7d5d70-3bf1-4b29-947a-0e7e175fabbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-348877-c6895b5c7-4m2kf" Oct 30 18:53:18.840992 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:18.840971 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd7d5d70-3bf1-4b29-947a-0e7e175fabbb-maas-api-tls\") pod \"maas-api-e2e-ipp-b-348877-c6895b5c7-4m2kf\" (UID: \"bd7d5d70-3bf1-4b29-947a-0e7e175fabbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-348877-c6895b5c7-4m2kf" Oct 30 18:53:18.841255 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:18.841238 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd7d5d70-3bf1-4b29-947a-0e7e175fabbb-metrics-certs\") pod \"maas-api-e2e-ipp-b-348877-c6895b5c7-4m2kf\" (UID: \"bd7d5d70-3bf1-4b29-947a-0e7e175fabbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-348877-c6895b5c7-4m2kf" Oct 30 18:53:18.853749 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:18.853730 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdzkt\" (UniqueName: \"kubernetes.io/projected/bd7d5d70-3bf1-4b29-947a-0e7e175fabbb-kube-api-access-tdzkt\") pod \"maas-api-e2e-ipp-b-348877-c6895b5c7-4m2kf\" (UID: \"bd7d5d70-3bf1-4b29-947a-0e7e175fabbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-348877-c6895b5c7-4m2kf" Oct 30 18:53:18.900850 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:18.900617 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-348877-c6895b5c7-4m2kf" Oct 30 18:53:18.950187 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:18.944936 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:19.162202 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:19.161942 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:19.227510 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:19.227480 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-348877-c6895b5c7-4m2kf"] Oct 30 18:53:19.229545 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:53:19.229514 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd7d5d70_3bf1_4b29_947a_0e7e175fabbb.slice/crio-de43292324e24ca08130095410d8a7eace774241db14ee8ab4dfbd5174ee094c WatchSource:0}: Error finding container de43292324e24ca08130095410d8a7eace774241db14ee8ab4dfbd5174ee094c: Status 404 returned error can't find the container with id de43292324e24ca08130095410d8a7eace774241db14ee8ab4dfbd5174ee094c Oct 30 18:53:19.270004 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:19.269980 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-348877-c6895b5c7-4m2kf" event={"ID":"bd7d5d70-3bf1-4b29-947a-0e7e175fabbb","Type":"ContainerStarted","Data":"de43292324e24ca08130095410d8a7eace774241db14ee8ab4dfbd5174ee094c"} Oct 30 18:53:20.108638 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:20.108551 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:20.277924 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:20.277892 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-348877-c6895b5c7-4m2kf" event={"ID":"bd7d5d70-3bf1-4b29-947a-0e7e175fabbb","Type":"ContainerStarted","Data":"df11d1251fe3fb1eb30520068099bad126a4b8745166f4032ea5147fde8cd8ad"} Oct 30 18:53:20.278109 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:20.278003 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-348877-c6895b5c7-4m2kf" Oct 30 18:53:20.316106 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:20.315014 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-348877-c6895b5c7-4m2kf" podStartSLOduration=1.7213804320000001 podStartE2EDuration="2.314988599s" totalImagesPullingTime="593.608167ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:53:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:53:20.309087896 +0000 UTC m=+1167.838119033" watchObservedRunningTime="2026-10-30 18:53:20.314988599 +0000 UTC m=+1167.844019738" Oct 30 18:53:21.367139 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:21.367106 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:21.553892 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:21.553778 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:21.752221 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:21.750740 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:21.977224 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:21.970214 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:22.420866 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:22.420801 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:23.515518 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:23.515480 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:23.935097 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:53:23.935065 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 18:53:24.600537 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:24.600497 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:24.713167 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:24.713111 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:25.084286 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:25.084254 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:25.882235 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:25.882198 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:26.291621 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:26.291553 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-348877-c6895b5c7-4m2kf" Oct 30 18:53:27.281742 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:27.281697 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:28.077723 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:28.077689 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:29.025805 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:29.025770 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:29.803603 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:29.803544 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:30.640887 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:30.640845 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:31.612520 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:31.612489 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:32.715922 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:32.715887 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:32.860431 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:32.860361 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:33.010935 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:33.010856 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:33.214650 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:33.214619 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:33.719554 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:33.719467 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:34.348039 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:34.348007 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-45f649-7577cd8b64-w75hv"] Oct 30 18:53:34.348570 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:34.348536 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45f649-7577cd8b64-w75hv" podUID="364b132c-231b-4108-9808-7ebbef4a9fbc" containerName="maas-api" containerID="cri-o://0da851fccf82b4be86a367340005bdfda38d52b2b42ff7c49dffd6e10a42420b" gracePeriod=30 Oct 30 18:53:34.409821 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:34.409790 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:34.428205 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:34.426280 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-45f649-6c66fcd59b-mv59d"] Oct 30 18:53:34.428205 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:34.426507 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-45f649-6c66fcd59b-mv59d" podUID="d3a1f960-c8df-46f6-b3a7-9b5020863765" containerName="payload-processing" containerID="cri-o://655f00e7b6e35a7f3a28a0e14f39fb2afea0e2d03e9c2fb488847a584ca2a5e4" gracePeriod=30 Oct 30 18:53:34.516168 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:34.512605 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:34.829568 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:34.829532 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:34.933976 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:34.933950 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45f649-7577cd8b64-w75hv" Oct 30 18:53:34.947072 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:34.947045 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:34.988006 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:34.987979 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-45f649-6c66fcd59b-mv59d" Oct 30 18:53:35.036506 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:35.036476 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/364b132c-231b-4108-9808-7ebbef4a9fbc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/364b132c-231b-4108-9808-7ebbef4a9fbc-maas-api-tls\") pod \"364b132c-231b-4108-9808-7ebbef4a9fbc\" (UID: \"364b132c-231b-4108-9808-7ebbef4a9fbc\") " Oct 30 18:53:35.036665 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:35.036578 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/364b132c-231b-4108-9808-7ebbef4a9fbc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/364b132c-231b-4108-9808-7ebbef4a9fbc-metrics-certs\") pod \"364b132c-231b-4108-9808-7ebbef4a9fbc\" (UID: \"364b132c-231b-4108-9808-7ebbef4a9fbc\") " Oct 30 18:53:35.036730 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:35.036670 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/364b132c-231b-4108-9808-7ebbef4a9fbc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/364b132c-231b-4108-9808-7ebbef4a9fbc-openshift-service-ca\") pod \"364b132c-231b-4108-9808-7ebbef4a9fbc\" (UID: \"364b132c-231b-4108-9808-7ebbef4a9fbc\") " Oct 30 18:53:35.036787 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:35.036770 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/364b132c-231b-4108-9808-7ebbef4a9fbc-kube-api-access-kjh4v\" (UniqueName: \"kubernetes.io/projected/364b132c-231b-4108-9808-7ebbef4a9fbc-kube-api-access-kjh4v\") pod \"364b132c-231b-4108-9808-7ebbef4a9fbc\" (UID: \"364b132c-231b-4108-9808-7ebbef4a9fbc\") " Oct 30 18:53:35.037088 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:35.037042 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/364b132c-231b-4108-9808-7ebbef4a9fbc-openshift-service-ca" pod "364b132c-231b-4108-9808-7ebbef4a9fbc" (UID: "364b132c-231b-4108-9808-7ebbef4a9fbc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:53:35.037585 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:35.037562 6960 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/364b132c-231b-4108-9808-7ebbef4a9fbc-openshift-service-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:53:35.039749 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:35.039720 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/364b132c-231b-4108-9808-7ebbef4a9fbc-maas-api-tls" pod "364b132c-231b-4108-9808-7ebbef4a9fbc" (UID: "364b132c-231b-4108-9808-7ebbef4a9fbc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:53:35.039842 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:35.039770 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/364b132c-231b-4108-9808-7ebbef4a9fbc-kube-api-access-kjh4v" pod "364b132c-231b-4108-9808-7ebbef4a9fbc" (UID: "364b132c-231b-4108-9808-7ebbef4a9fbc"). InnerVolumeSpecName "kube-api-access-kjh4v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:53:35.040333 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:35.040311 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/364b132c-231b-4108-9808-7ebbef4a9fbc-metrics-certs" pod "364b132c-231b-4108-9808-7ebbef4a9fbc" (UID: "364b132c-231b-4108-9808-7ebbef4a9fbc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:53:35.138893 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:35.138851 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3a1f960-c8df-46f6-b3a7-9b5020863765-kube-api-access-s7ldz\" (UniqueName: \"kubernetes.io/projected/d3a1f960-c8df-46f6-b3a7-9b5020863765-kube-api-access-s7ldz\") pod \"d3a1f960-c8df-46f6-b3a7-9b5020863765\" (UID: \"d3a1f960-c8df-46f6-b3a7-9b5020863765\") " Oct 30 18:53:35.139293 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:35.139267 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3a1f960-c8df-46f6-b3a7-9b5020863765-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3a1f960-c8df-46f6-b3a7-9b5020863765-plugins-config-volume\") pod \"d3a1f960-c8df-46f6-b3a7-9b5020863765\" (UID: \"d3a1f960-c8df-46f6-b3a7-9b5020863765\") " Oct 30 18:53:35.139772 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:35.139756 6960 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/364b132c-231b-4108-9808-7ebbef4a9fbc-maas-api-tls\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:53:35.139834 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:35.139775 6960 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/364b132c-231b-4108-9808-7ebbef4a9fbc-metrics-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:53:35.139834 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:35.139786 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjh4v\" (UniqueName: \"kubernetes.io/projected/364b132c-231b-4108-9808-7ebbef4a9fbc-kube-api-access-kjh4v\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:53:35.139906 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:35.139864 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3a1f960-c8df-46f6-b3a7-9b5020863765-plugins-config-volume" pod "d3a1f960-c8df-46f6-b3a7-9b5020863765" (UID: "d3a1f960-c8df-46f6-b3a7-9b5020863765"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:53:35.141261 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:35.141234 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3a1f960-c8df-46f6-b3a7-9b5020863765-kube-api-access-s7ldz" pod "d3a1f960-c8df-46f6-b3a7-9b5020863765" (UID: "d3a1f960-c8df-46f6-b3a7-9b5020863765"). InnerVolumeSpecName "kube-api-access-s7ldz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:53:35.236558 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:35.236476 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:35.241021 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:35.240985 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7ldz\" (UniqueName: \"kubernetes.io/projected/d3a1f960-c8df-46f6-b3a7-9b5020863765-kube-api-access-s7ldz\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:53:35.241021 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:35.241010 6960 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3a1f960-c8df-46f6-b3a7-9b5020863765-plugins-config-volume\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:53:35.398378 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:35.398344 6960 generic.go:356] "Generic (PLEG): container finished" podID="d3a1f960-c8df-46f6-b3a7-9b5020863765" containerID="655f00e7b6e35a7f3a28a0e14f39fb2afea0e2d03e9c2fb488847a584ca2a5e4" exitCode=0 Oct 30 18:53:35.398513 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:35.398395 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-45f649-6c66fcd59b-mv59d" Oct 30 18:53:35.398513 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:35.398429 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-45f649-6c66fcd59b-mv59d" event={"ID":"d3a1f960-c8df-46f6-b3a7-9b5020863765","Type":"ContainerDied","Data":"655f00e7b6e35a7f3a28a0e14f39fb2afea0e2d03e9c2fb488847a584ca2a5e4"} Oct 30 18:53:35.398513 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:35.398503 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-45f649-6c66fcd59b-mv59d" event={"ID":"d3a1f960-c8df-46f6-b3a7-9b5020863765","Type":"ContainerDied","Data":"5559b79a1f1af04a5094a8ed98723b6d8b0486f6ab3438812ef8cd1103c73aa3"} Oct 30 18:53:35.398694 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:35.398532 6960 scope.go:122] "RemoveContainer" containerID="655f00e7b6e35a7f3a28a0e14f39fb2afea0e2d03e9c2fb488847a584ca2a5e4" Oct 30 18:53:35.399845 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:35.399822 6960 generic.go:356] "Generic (PLEG): container finished" podID="364b132c-231b-4108-9808-7ebbef4a9fbc" containerID="0da851fccf82b4be86a367340005bdfda38d52b2b42ff7c49dffd6e10a42420b" exitCode=0 Oct 30 18:53:35.399920 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:35.399857 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45f649-7577cd8b64-w75hv" event={"ID":"364b132c-231b-4108-9808-7ebbef4a9fbc","Type":"ContainerDied","Data":"0da851fccf82b4be86a367340005bdfda38d52b2b42ff7c49dffd6e10a42420b"} Oct 30 18:53:35.399972 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:35.399918 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45f649-7577cd8b64-w75hv" event={"ID":"364b132c-231b-4108-9808-7ebbef4a9fbc","Type":"ContainerDied","Data":"d8df0a7b2acb26eb7a31c5bec7768e4a5c0212f309573992a74f7574c035e767"} Oct 30 18:53:35.399972 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:35.399867 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45f649-7577cd8b64-w75hv" Oct 30 18:53:35.411676 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:35.411657 6960 scope.go:122] "RemoveContainer" containerID="655f00e7b6e35a7f3a28a0e14f39fb2afea0e2d03e9c2fb488847a584ca2a5e4" Oct 30 18:53:35.411933 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:53:35.411914 6960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"655f00e7b6e35a7f3a28a0e14f39fb2afea0e2d03e9c2fb488847a584ca2a5e4\": container with ID starting with 655f00e7b6e35a7f3a28a0e14f39fb2afea0e2d03e9c2fb488847a584ca2a5e4 not found: ID does not exist" containerID="655f00e7b6e35a7f3a28a0e14f39fb2afea0e2d03e9c2fb488847a584ca2a5e4" Oct 30 18:53:35.412006 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:35.411941 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"655f00e7b6e35a7f3a28a0e14f39fb2afea0e2d03e9c2fb488847a584ca2a5e4"} err="failed to get container status \"655f00e7b6e35a7f3a28a0e14f39fb2afea0e2d03e9c2fb488847a584ca2a5e4\": rpc error: code = NotFound desc = could not find container \"655f00e7b6e35a7f3a28a0e14f39fb2afea0e2d03e9c2fb488847a584ca2a5e4\": container with ID starting with 655f00e7b6e35a7f3a28a0e14f39fb2afea0e2d03e9c2fb488847a584ca2a5e4 not found: ID does not exist" Oct 30 18:53:35.412006 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:35.411958 6960 scope.go:122] "RemoveContainer" containerID="0da851fccf82b4be86a367340005bdfda38d52b2b42ff7c49dffd6e10a42420b" Oct 30 18:53:35.425117 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:35.425096 6960 scope.go:122] "RemoveContainer" containerID="0da851fccf82b4be86a367340005bdfda38d52b2b42ff7c49dffd6e10a42420b" Oct 30 18:53:35.425519 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:53:35.425488 6960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0da851fccf82b4be86a367340005bdfda38d52b2b42ff7c49dffd6e10a42420b\": container with ID starting with 0da851fccf82b4be86a367340005bdfda38d52b2b42ff7c49dffd6e10a42420b not found: ID does not exist" containerID="0da851fccf82b4be86a367340005bdfda38d52b2b42ff7c49dffd6e10a42420b" Oct 30 18:53:35.425623 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:35.425527 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0da851fccf82b4be86a367340005bdfda38d52b2b42ff7c49dffd6e10a42420b"} err="failed to get container status \"0da851fccf82b4be86a367340005bdfda38d52b2b42ff7c49dffd6e10a42420b\": rpc error: code = NotFound desc = could not find container \"0da851fccf82b4be86a367340005bdfda38d52b2b42ff7c49dffd6e10a42420b\": container with ID starting with 0da851fccf82b4be86a367340005bdfda38d52b2b42ff7c49dffd6e10a42420b not found: ID does not exist" Oct 30 18:53:35.428646 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:35.428604 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-45f649-6c66fcd59b-mv59d"] Oct 30 18:53:35.431674 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:35.431657 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-45f649-6c66fcd59b-mv59d"] Oct 30 18:53:35.445913 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:35.445886 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-45f649-7577cd8b64-w75hv"] Oct 30 18:53:35.452587 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:35.452565 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-45f649-7577cd8b64-w75hv"] Oct 30 18:53:35.492173 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:35.492087 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:36.336218 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:36.336190 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:36.638401 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:36.638305 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="364b132c-231b-4108-9808-7ebbef4a9fbc" path="/var/lib/kubelet/pods/364b132c-231b-4108-9808-7ebbef4a9fbc/volumes" Oct 30 18:53:36.639090 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:36.639057 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3a1f960-c8df-46f6-b3a7-9b5020863765" path="/var/lib/kubelet/pods/d3a1f960-c8df-46f6-b3a7-9b5020863765/volumes" Oct 30 18:53:37.339641 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:37.339607 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:37.949358 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:37.949324 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8fd4ed-57674cb595-sdrlj"] Oct 30 18:53:38.019531 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:38.019497 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:38.156431 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:38.156385 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:38.300222 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:38.292429 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:38.433307 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:38.432990 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8fd4ed-57674cb595-sdrlj" podUID="358545ad-9045-403a-9d72-aa4297431ba1" containerName="maas-api" containerID="cri-o://85ff683edaaa9f1c13e994d029465ab53188b403f7e84e846bbb91d7ea22f6fd" gracePeriod=30 Oct 30 18:53:38.474623 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:38.474580 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:38.603241 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:38.603211 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:38.766458 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:38.766433 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8fd4ed-57674cb595-sdrlj" Oct 30 18:53:38.809957 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:38.809866 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/358545ad-9045-403a-9d72-aa4297431ba1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/358545ad-9045-403a-9d72-aa4297431ba1-maas-api-tls\") pod \"358545ad-9045-403a-9d72-aa4297431ba1\" (UID: \"358545ad-9045-403a-9d72-aa4297431ba1\") " Oct 30 18:53:38.809957 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:38.809918 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/358545ad-9045-403a-9d72-aa4297431ba1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/358545ad-9045-403a-9d72-aa4297431ba1-openshift-service-ca\") pod \"358545ad-9045-403a-9d72-aa4297431ba1\" (UID: \"358545ad-9045-403a-9d72-aa4297431ba1\") " Oct 30 18:53:38.810227 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:38.809983 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/358545ad-9045-403a-9d72-aa4297431ba1-kube-api-access-7w5dg\" (UniqueName: \"kubernetes.io/projected/358545ad-9045-403a-9d72-aa4297431ba1-kube-api-access-7w5dg\") pod \"358545ad-9045-403a-9d72-aa4297431ba1\" (UID: \"358545ad-9045-403a-9d72-aa4297431ba1\") " Oct 30 18:53:38.810227 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:38.810050 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/358545ad-9045-403a-9d72-aa4297431ba1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/358545ad-9045-403a-9d72-aa4297431ba1-metrics-certs\") pod \"358545ad-9045-403a-9d72-aa4297431ba1\" (UID: \"358545ad-9045-403a-9d72-aa4297431ba1\") " Oct 30 18:53:38.810800 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:38.810741 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/358545ad-9045-403a-9d72-aa4297431ba1-openshift-service-ca" pod "358545ad-9045-403a-9d72-aa4297431ba1" (UID: "358545ad-9045-403a-9d72-aa4297431ba1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:53:38.811465 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:38.811444 6960 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/358545ad-9045-403a-9d72-aa4297431ba1-openshift-service-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:53:38.813785 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:38.813757 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/358545ad-9045-403a-9d72-aa4297431ba1-kube-api-access-7w5dg" pod "358545ad-9045-403a-9d72-aa4297431ba1" (UID: "358545ad-9045-403a-9d72-aa4297431ba1"). InnerVolumeSpecName "kube-api-access-7w5dg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:53:38.814166 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:38.814104 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/358545ad-9045-403a-9d72-aa4297431ba1-metrics-certs" pod "358545ad-9045-403a-9d72-aa4297431ba1" (UID: "358545ad-9045-403a-9d72-aa4297431ba1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:53:38.815088 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:38.815064 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/358545ad-9045-403a-9d72-aa4297431ba1-maas-api-tls" pod "358545ad-9045-403a-9d72-aa4297431ba1" (UID: "358545ad-9045-403a-9d72-aa4297431ba1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:53:38.913262 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:38.913227 6960 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/358545ad-9045-403a-9d72-aa4297431ba1-maas-api-tls\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:53:38.913262 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:38.913252 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7w5dg\" (UniqueName: \"kubernetes.io/projected/358545ad-9045-403a-9d72-aa4297431ba1-kube-api-access-7w5dg\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:53:38.913262 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:38.913268 6960 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/358545ad-9045-403a-9d72-aa4297431ba1-metrics-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:53:38.993119 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:38.993070 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:39.440319 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:39.440285 6960 generic.go:356] "Generic (PLEG): container finished" podID="358545ad-9045-403a-9d72-aa4297431ba1" containerID="85ff683edaaa9f1c13e994d029465ab53188b403f7e84e846bbb91d7ea22f6fd" exitCode=0 Oct 30 18:53:39.440777 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:39.440334 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8fd4ed-57674cb595-sdrlj" Oct 30 18:53:39.440777 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:39.440357 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8fd4ed-57674cb595-sdrlj" event={"ID":"358545ad-9045-403a-9d72-aa4297431ba1","Type":"ContainerDied","Data":"85ff683edaaa9f1c13e994d029465ab53188b403f7e84e846bbb91d7ea22f6fd"} Oct 30 18:53:39.440777 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:39.440402 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8fd4ed-57674cb595-sdrlj" event={"ID":"358545ad-9045-403a-9d72-aa4297431ba1","Type":"ContainerDied","Data":"3eb2dbb73519a640501cc0ba739925981e44eb597f4a071e3b9598ee09b37990"} Oct 30 18:53:39.440777 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:39.440425 6960 scope.go:122] "RemoveContainer" containerID="85ff683edaaa9f1c13e994d029465ab53188b403f7e84e846bbb91d7ea22f6fd" Oct 30 18:53:39.452088 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:39.452070 6960 scope.go:122] "RemoveContainer" containerID="85ff683edaaa9f1c13e994d029465ab53188b403f7e84e846bbb91d7ea22f6fd" Oct 30 18:53:39.452388 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:53:39.452368 6960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85ff683edaaa9f1c13e994d029465ab53188b403f7e84e846bbb91d7ea22f6fd\": container with ID starting with 85ff683edaaa9f1c13e994d029465ab53188b403f7e84e846bbb91d7ea22f6fd not found: ID does not exist" containerID="85ff683edaaa9f1c13e994d029465ab53188b403f7e84e846bbb91d7ea22f6fd" Oct 30 18:53:39.452463 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:39.452402 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85ff683edaaa9f1c13e994d029465ab53188b403f7e84e846bbb91d7ea22f6fd"} err="failed to get container status \"85ff683edaaa9f1c13e994d029465ab53188b403f7e84e846bbb91d7ea22f6fd\": rpc error: code = NotFound desc = could not find container \"85ff683edaaa9f1c13e994d029465ab53188b403f7e84e846bbb91d7ea22f6fd\": container with ID starting with 85ff683edaaa9f1c13e994d029465ab53188b403f7e84e846bbb91d7ea22f6fd not found: ID does not exist" Oct 30 18:53:39.466271 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:39.466252 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8fd4ed-57674cb595-sdrlj"] Oct 30 18:53:39.470450 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:39.470432 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8fd4ed-57674cb595-sdrlj"] Oct 30 18:53:39.486721 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:39.486693 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:40.377680 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:40.377641 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:40.593079 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:40.593050 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="358545ad-9045-403a-9d72-aa4297431ba1" path="/var/lib/kubelet/pods/358545ad-9045-403a-9d72-aa4297431ba1/volumes" Oct 30 18:53:41.074476 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:41.074445 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:41.180993 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:41.180955 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:41.594389 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:41.594357 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:41.654210 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:41.654181 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ecef08-7bcb64d759-bqlt4"] Oct 30 18:53:41.654501 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:41.654467 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ecef08-7bcb64d759-bqlt4" podUID="4a18b92c-2dd3-4a63-8eb4-0fdb4f34e470" containerName="payload-pre-processing" containerID="cri-o://8c4e66a797da4f97eb8f7420fc931535fff1a51814b872121efe18f089cd149e" gracePeriod=30 Oct 30 18:53:41.762709 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:41.762671 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:41.890790 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:41.890660 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:42.030220 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:42.025350 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:42.121196 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:42.121170 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ecef08-7bcb64d759-bqlt4" Oct 30 18:53:42.145391 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:42.145298 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:42.205574 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:42.205529 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a18b92c-2dd3-4a63-8eb4-0fdb4f34e470-kube-api-access-hx46q\" (UniqueName: \"kubernetes.io/projected/4a18b92c-2dd3-4a63-8eb4-0fdb4f34e470-kube-api-access-hx46q\") pod \"4a18b92c-2dd3-4a63-8eb4-0fdb4f34e470\" (UID: \"4a18b92c-2dd3-4a63-8eb4-0fdb4f34e470\") " Oct 30 18:53:42.205746 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:42.205714 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a18b92c-2dd3-4a63-8eb4-0fdb4f34e470-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a18b92c-2dd3-4a63-8eb4-0fdb4f34e470-plugins-config-volume\") pod \"4a18b92c-2dd3-4a63-8eb4-0fdb4f34e470\" (UID: \"4a18b92c-2dd3-4a63-8eb4-0fdb4f34e470\") " Oct 30 18:53:42.206280 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:42.206103 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a18b92c-2dd3-4a63-8eb4-0fdb4f34e470-plugins-config-volume" pod "4a18b92c-2dd3-4a63-8eb4-0fdb4f34e470" (UID: "4a18b92c-2dd3-4a63-8eb4-0fdb4f34e470"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:53:42.206740 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:42.206716 6960 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a18b92c-2dd3-4a63-8eb4-0fdb4f34e470-plugins-config-volume\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:53:42.208093 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:42.208069 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a18b92c-2dd3-4a63-8eb4-0fdb4f34e470-kube-api-access-hx46q" pod "4a18b92c-2dd3-4a63-8eb4-0fdb4f34e470" (UID: "4a18b92c-2dd3-4a63-8eb4-0fdb4f34e470"). InnerVolumeSpecName "kube-api-access-hx46q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:53:42.308694 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:42.308656 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hx46q\" (UniqueName: \"kubernetes.io/projected/4a18b92c-2dd3-4a63-8eb4-0fdb4f34e470-kube-api-access-hx46q\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:53:42.470731 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:42.470636 6960 generic.go:356] "Generic (PLEG): container finished" podID="4a18b92c-2dd3-4a63-8eb4-0fdb4f34e470" containerID="8c4e66a797da4f97eb8f7420fc931535fff1a51814b872121efe18f089cd149e" exitCode=0 Oct 30 18:53:42.470731 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:42.470694 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ecef08-7bcb64d759-bqlt4" Oct 30 18:53:42.470977 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:42.470726 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ecef08-7bcb64d759-bqlt4" event={"ID":"4a18b92c-2dd3-4a63-8eb4-0fdb4f34e470","Type":"ContainerDied","Data":"8c4e66a797da4f97eb8f7420fc931535fff1a51814b872121efe18f089cd149e"} Oct 30 18:53:42.470977 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:42.470808 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ecef08-7bcb64d759-bqlt4" event={"ID":"4a18b92c-2dd3-4a63-8eb4-0fdb4f34e470","Type":"ContainerDied","Data":"01a25dfa049d1588f69d06cf1887a2d392ebc459b2a4d3a549381e348059046f"} Oct 30 18:53:42.470977 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:42.470839 6960 scope.go:122] "RemoveContainer" containerID="8c4e66a797da4f97eb8f7420fc931535fff1a51814b872121efe18f089cd149e" Oct 30 18:53:42.482129 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:42.482110 6960 scope.go:122] "RemoveContainer" containerID="8c4e66a797da4f97eb8f7420fc931535fff1a51814b872121efe18f089cd149e" Oct 30 18:53:42.482489 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:53:42.482470 6960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c4e66a797da4f97eb8f7420fc931535fff1a51814b872121efe18f089cd149e\": container with ID starting with 8c4e66a797da4f97eb8f7420fc931535fff1a51814b872121efe18f089cd149e not found: ID does not exist" containerID="8c4e66a797da4f97eb8f7420fc931535fff1a51814b872121efe18f089cd149e" Oct 30 18:53:42.482540 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:42.482507 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c4e66a797da4f97eb8f7420fc931535fff1a51814b872121efe18f089cd149e"} err="failed to get container status \"8c4e66a797da4f97eb8f7420fc931535fff1a51814b872121efe18f089cd149e\": rpc error: code = NotFound desc = could not find container \"8c4e66a797da4f97eb8f7420fc931535fff1a51814b872121efe18f089cd149e\": container with ID starting with 8c4e66a797da4f97eb8f7420fc931535fff1a51814b872121efe18f089cd149e not found: ID does not exist" Oct 30 18:53:42.497915 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:42.497862 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ecef08-7bcb64d759-bqlt4"] Oct 30 18:53:42.501522 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:42.501502 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ecef08-7bcb64d759-bqlt4"] Oct 30 18:53:42.544982 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:42.544954 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:42.594231 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:42.594202 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a18b92c-2dd3-4a63-8eb4-0fdb4f34e470" path="/var/lib/kubelet/pods/4a18b92c-2dd3-4a63-8eb4-0fdb4f34e470/volumes" Oct 30 18:53:43.028277 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:43.028245 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:43.675175 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:43.675126 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:43.785066 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:43.784935 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:44.000938 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:44.000840 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:44.178169 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:44.169705 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:44.918780 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:44.918737 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:45.702155 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:45.702104 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:46.382171 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:46.379523 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:46.511655 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:46.511629 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:47.708261 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:47.708233 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:48.673465 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:48.673342 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:48.795200 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:48.795166 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:49.131573 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:49.131542 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:49.850056 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:49.850024 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:50.492923 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:50.492886 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:50.624883 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:50.624845 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:50.832551 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:50.832515 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:51.047758 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:51.047722 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:51.293669 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:51.293584 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:51.476615 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:51.476583 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:51.747941 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:51.747892 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:52.080068 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:52.080033 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:52.231760 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:52.231708 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:52.402065 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:52.401969 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:53.056690 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:53.056645 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:53.202179 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:53.202112 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:53.510693 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:53.510656 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:53.680172 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:53.680004 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:53.937364 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:53:53.937330 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 18:53:54.206416 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:54.206327 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:55.319079 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:55.319041 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:55.736918 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:55.736822 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:55.875777 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:55.875747 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:56.492663 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:56.492626 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:56.967793 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:56.967760 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:57.124122 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:57.124082 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:57.679576 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:57.679540 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:57.894993 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:57.894962 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:58.902265 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:58.902237 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:59.039791 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:59.039759 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:59.917560 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:59.917512 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:59.923137 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:59.923112 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-fa7b20-7ff6f5cfdf-2n8cn"] Oct 30 18:53:59.923722 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:59.923708 6960 cpu_manager.go:414] "containerMap: removing container" podUID="4a18b92c-2dd3-4a63-8eb4-0fdb4f34e470" containerName="payload-pre-processing" Oct 30 18:53:59.923778 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:59.923723 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a18b92c-2dd3-4a63-8eb4-0fdb4f34e470" containerName="payload-pre-processing" Oct 30 18:53:59.923778 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:59.923737 6960 cpu_manager.go:414] "containerMap: removing container" podUID="358545ad-9045-403a-9d72-aa4297431ba1" containerName="maas-api" Oct 30 18:53:59.923778 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:59.923742 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="358545ad-9045-403a-9d72-aa4297431ba1" containerName="maas-api" Oct 30 18:53:59.923778 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:59.923760 6960 cpu_manager.go:414] "containerMap: removing container" podUID="d3a1f960-c8df-46f6-b3a7-9b5020863765" containerName="payload-processing" Oct 30 18:53:59.923778 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:59.923765 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3a1f960-c8df-46f6-b3a7-9b5020863765" containerName="payload-processing" Oct 30 18:53:59.923932 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:59.923784 6960 cpu_manager.go:414] "containerMap: removing container" podUID="364b132c-231b-4108-9808-7ebbef4a9fbc" containerName="maas-api" Oct 30 18:53:59.923932 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:59.923789 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="364b132c-231b-4108-9808-7ebbef4a9fbc" containerName="maas-api" Oct 30 18:53:59.923932 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:59.923877 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3a1f960-c8df-46f6-b3a7-9b5020863765" containerName="payload-processing" Oct 30 18:53:59.923932 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:59.923888 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="364b132c-231b-4108-9808-7ebbef4a9fbc" containerName="maas-api" Oct 30 18:53:59.923932 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:59.923899 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a18b92c-2dd3-4a63-8eb4-0fdb4f34e470" containerName="payload-pre-processing" Oct 30 18:53:59.923932 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:59.923905 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="358545ad-9045-403a-9d72-aa4297431ba1" containerName="maas-api" Oct 30 18:53:59.929228 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:59.929208 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fa7b20-7ff6f5cfdf-2n8cn" Oct 30 18:53:59.942343 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:59.942299 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-fa7b20-serving-cert\"" Oct 30 18:53:59.953013 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:53:59.952985 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-fa7b20-7ff6f5cfdf-2n8cn"] Oct 30 18:54:00.042022 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:00.041951 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:00.089042 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:00.089009 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvkcx\" (UniqueName: \"kubernetes.io/projected/95c445fe-9636-49bd-b910-8ea961f962ce-kube-api-access-dvkcx\") pod \"maas-api-e2e-praxis-a-fa7b20-7ff6f5cfdf-2n8cn\" (UID: \"95c445fe-9636-49bd-b910-8ea961f962ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fa7b20-7ff6f5cfdf-2n8cn" Oct 30 18:54:00.089240 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:00.089079 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95c445fe-9636-49bd-b910-8ea961f962ce-maas-api-tls\") pod \"maas-api-e2e-praxis-a-fa7b20-7ff6f5cfdf-2n8cn\" (UID: \"95c445fe-9636-49bd-b910-8ea961f962ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fa7b20-7ff6f5cfdf-2n8cn" Oct 30 18:54:00.089356 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:00.089335 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95c445fe-9636-49bd-b910-8ea961f962ce-metrics-certs\") pod \"maas-api-e2e-praxis-a-fa7b20-7ff6f5cfdf-2n8cn\" (UID: \"95c445fe-9636-49bd-b910-8ea961f962ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fa7b20-7ff6f5cfdf-2n8cn" Oct 30 18:54:00.089476 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:00.089455 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95c445fe-9636-49bd-b910-8ea961f962ce-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-fa7b20-7ff6f5cfdf-2n8cn\" (UID: \"95c445fe-9636-49bd-b910-8ea961f962ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fa7b20-7ff6f5cfdf-2n8cn" Oct 30 18:54:00.155673 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:00.155631 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:00.191137 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:00.191070 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvkcx\" (UniqueName: \"kubernetes.io/projected/95c445fe-9636-49bd-b910-8ea961f962ce-kube-api-access-dvkcx\") pod \"maas-api-e2e-praxis-a-fa7b20-7ff6f5cfdf-2n8cn\" (UID: \"95c445fe-9636-49bd-b910-8ea961f962ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fa7b20-7ff6f5cfdf-2n8cn" Oct 30 18:54:00.191298 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:00.191175 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95c445fe-9636-49bd-b910-8ea961f962ce-maas-api-tls\") pod \"maas-api-e2e-praxis-a-fa7b20-7ff6f5cfdf-2n8cn\" (UID: \"95c445fe-9636-49bd-b910-8ea961f962ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fa7b20-7ff6f5cfdf-2n8cn" Oct 30 18:54:00.191349 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:00.191337 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95c445fe-9636-49bd-b910-8ea961f962ce-metrics-certs\") pod \"maas-api-e2e-praxis-a-fa7b20-7ff6f5cfdf-2n8cn\" (UID: \"95c445fe-9636-49bd-b910-8ea961f962ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fa7b20-7ff6f5cfdf-2n8cn" Oct 30 18:54:00.191409 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:00.191397 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95c445fe-9636-49bd-b910-8ea961f962ce-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-fa7b20-7ff6f5cfdf-2n8cn\" (UID: \"95c445fe-9636-49bd-b910-8ea961f962ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fa7b20-7ff6f5cfdf-2n8cn" Oct 30 18:54:00.192036 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:00.191966 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95c445fe-9636-49bd-b910-8ea961f962ce-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-fa7b20-7ff6f5cfdf-2n8cn\" (UID: \"95c445fe-9636-49bd-b910-8ea961f962ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fa7b20-7ff6f5cfdf-2n8cn" Oct 30 18:54:00.193732 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:00.193714 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95c445fe-9636-49bd-b910-8ea961f962ce-metrics-certs\") pod \"maas-api-e2e-praxis-a-fa7b20-7ff6f5cfdf-2n8cn\" (UID: \"95c445fe-9636-49bd-b910-8ea961f962ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fa7b20-7ff6f5cfdf-2n8cn" Oct 30 18:54:00.193912 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:00.193888 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95c445fe-9636-49bd-b910-8ea961f962ce-maas-api-tls\") pod \"maas-api-e2e-praxis-a-fa7b20-7ff6f5cfdf-2n8cn\" (UID: \"95c445fe-9636-49bd-b910-8ea961f962ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fa7b20-7ff6f5cfdf-2n8cn" Oct 30 18:54:00.215422 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:00.215388 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvkcx\" (UniqueName: \"kubernetes.io/projected/95c445fe-9636-49bd-b910-8ea961f962ce-kube-api-access-dvkcx\") pod \"maas-api-e2e-praxis-a-fa7b20-7ff6f5cfdf-2n8cn\" (UID: \"95c445fe-9636-49bd-b910-8ea961f962ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fa7b20-7ff6f5cfdf-2n8cn" Oct 30 18:54:00.247476 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:00.247445 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fa7b20-7ff6f5cfdf-2n8cn" Oct 30 18:54:00.535957 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:00.535927 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-fa7b20-7ff6f5cfdf-2n8cn"] Oct 30 18:54:00.536287 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:54:00.536264 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95c445fe_9636_49bd_b910_8ea961f962ce.slice/crio-53f8c06b3d0a8d3e5a36c922a3d07b3625e8d3a4fcc60e35a2086991fbd72e0c WatchSource:0}: Error finding container 53f8c06b3d0a8d3e5a36c922a3d07b3625e8d3a4fcc60e35a2086991fbd72e0c: Status 404 returned error can't find the container with id 53f8c06b3d0a8d3e5a36c922a3d07b3625e8d3a4fcc60e35a2086991fbd72e0c Oct 30 18:54:00.734261 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:00.734217 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fa7b20-7ff6f5cfdf-2n8cn" event={"ID":"95c445fe-9636-49bd-b910-8ea961f962ce","Type":"ContainerStarted","Data":"53f8c06b3d0a8d3e5a36c922a3d07b3625e8d3a4fcc60e35a2086991fbd72e0c"} Oct 30 18:54:00.902442 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:00.902408 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:01.742100 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:01.742068 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fa7b20-7ff6f5cfdf-2n8cn" event={"ID":"95c445fe-9636-49bd-b910-8ea961f962ce","Type":"ContainerStarted","Data":"d6cc0540302d4b26a162a04cd6fded0fe075906fe2a76a6719bfc54109f03854"} Oct 30 18:54:01.742535 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:01.742180 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fa7b20-7ff6f5cfdf-2n8cn" Oct 30 18:54:01.764705 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:01.764666 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fa7b20-7ff6f5cfdf-2n8cn" podStartSLOduration=2.3037274930000002 podStartE2EDuration="2.764654337s" totalImagesPullingTime="460.926844ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:53:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:54:01.760978347 +0000 UTC m=+1209.290009486" watchObservedRunningTime="2026-10-30 18:54:01.764654337 +0000 UTC m=+1209.293685475" Oct 30 18:54:02.015354 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:02.015262 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:02.804836 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:02.804794 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:03.598886 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:03.598848 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:04.622928 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:04.622885 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:04.740621 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:04.740586 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:04.869213 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:04.869173 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:05.001220 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:04.999350 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:05.663166 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:05.662446 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:06.777448 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:06.777408 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:06.907473 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:06.907435 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:07.754323 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:07.754297 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fa7b20-7ff6f5cfdf-2n8cn" Oct 30 18:54:07.819465 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:07.819431 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:08.725371 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:08.725336 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:09.608886 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:09.608850 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:10.638266 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:10.638235 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:11.968944 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:11.968913 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:12.684025 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:12.683994 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:13.091382 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:13.091335 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:13.924113 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:13.924079 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:14.834600 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:14.834563 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:15.747790 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:15.747758 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:15.878703 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:15.878669 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:16.710580 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:16.710547 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:17.134700 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:17.134668 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:17.803297 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:17.803256 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:18.204333 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:18.204286 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:18.459975 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:18.459873 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:19.319032 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:19.318997 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:20.257499 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:20.257468 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:20.938036 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:20.938006 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:21.248009 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:21.247918 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:21.719314 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:21.719275 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:21.868734 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:21.868698 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:22.244983 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:22.244950 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:22.367233 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:22.367189 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:22.912172 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:22.905833 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:23.118922 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:23.118876 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:23.276306 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:23.276226 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:23.476495 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:23.476461 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:23.937454 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:54:23.937416 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 18:54:24.038966 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:24.038930 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:24.382791 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:24.382756 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:25.102459 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:25.102298 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:25.433743 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:25.433654 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:25.640173 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:25.639372 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:26.333983 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:26.333950 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:27.456804 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:27.456771 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:27.845952 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:27.845904 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:27.965415 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:27.965382 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:28.465225 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:28.465196 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:28.607172 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:28.606999 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:29.114445 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:29.114366 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:29.314611 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:29.314571 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:29.998673 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:29.998638 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:30.105681 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:30.105627 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:30.253935 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:30.253852 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:30.378521 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:30.378491 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:31.187980 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:31.187922 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:32.008881 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:32.008845 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:32.815633 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:32.815602 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:33.832018 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:33.831980 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:33.943805 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:33.943772 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:34.748580 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:34.748538 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:35.704226 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:35.704187 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:36.588414 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:36.588376 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:37.423691 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:37.423652 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:37.854405 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:37.854372 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:38.825563 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:38.825526 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:39.779604 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:39.779565 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:39.900635 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:39.900594 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:40.032533 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:40.032411 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:40.725921 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:40.725892 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:40.841642 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:40.841608 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:40.969265 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:40.969223 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:41.703906 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:41.703868 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:42.502417 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:42.502378 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:43.432651 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:43.432605 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:44.384257 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:44.384224 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:45.272231 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:45.272202 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:46.087132 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:46.087084 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:46.882534 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:46.882492 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:47.771169 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:47.767337 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:48.873053 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:48.873007 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:49.804466 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:49.804425 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:50.832554 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:50.832516 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:51.537026 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:51.536976 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:51.711549 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:51.711511 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:52.159958 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:52.159921 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:52.500016 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:52.499931 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:53.559830 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:53.559793 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:53.937104 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:54:53.937021 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 18:54:54.793640 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:54.793605 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:55.833592 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:55.833559 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:56.608931 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:56.608897 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:57.309951 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:57.309915 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:57.746931 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:57.746815 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:58.357592 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:58.357552 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:58.711864 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:58.711770 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:58.884004 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:58.883961 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:59.504563 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:54:59.504494 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:00.494057 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:55:00.494021 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:01.618728 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:55:01.618687 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:03.249822 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:55:03.249791 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:04.724212 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:55:04.724122 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:05.937230 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:55:05.937094 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:06.805379 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:55:06.805342 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:07.707397 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:55:07.707366 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:08.521670 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:55:08.521632 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:09.484417 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:55:09.484376 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:10.375078 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:55:10.375045 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:11.088126 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:55:11.088089 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:11.196208 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:55:11.196174 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:11.515276 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:55:11.515182 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:12.233253 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:55:12.233209 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:12.483131 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:55:12.483099 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:12.796516 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:55:12.796475 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:13.083664 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:55:13.083623 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:13.206474 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:55:13.206439 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:13.328692 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:55:13.328656 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:13.784773 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:55:13.784741 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:14.163651 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:55:14.163612 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:15.011861 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:55:15.011832 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:15.792830 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:55:15.792796 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:16.552816 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:55:16.552786 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:17.403958 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:55:17.403928 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:18.400697 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:55:18.400667 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:19.236355 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:55:19.236327 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:20.125434 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:55:20.125397 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:21.028137 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:55:21.028101 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:21.185615 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:55:21.185563 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:21.362684 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:55:21.362640 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:22.337335 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:55:22.337293 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:23.156489 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:55:23.156449 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:23.408726 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:55:23.408638 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:23.669188 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:55:23.669070 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:23.943135 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:55:23.943042 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 18:55:24.026186 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:55:24.026136 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:24.217504 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:55:24.217434 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:25.012812 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:55:25.012785 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:26.091077 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:55:26.091040 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:26.901523 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:55:26.901488 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:27.679559 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:55:27.679527 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:28.594893 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:55:28.594856 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:29.440487 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:55:29.440448 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:30.428160 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:55:30.428112 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:30.699935 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:55:30.699841 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:30.837458 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:55:30.837424 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:31.159729 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:55:31.159697 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:31.750015 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:55:31.749962 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:32.095308 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:55:32.095253 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:32.348192 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:55:32.348084 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:33.000371 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:55:33.000335 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:33.808602 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:55:33.808557 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:34.010216 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:55:34.008690 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:34.171651 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:55:34.171577 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:34.351678 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:55:34.351643 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:34.907821 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:55:34.907785 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:35.594330 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:55:35.594299 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:36.942991 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:55:36.942955 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:37.822085 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:55:37.822046 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:38.714471 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:55:38.714418 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:39.678763 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:55:39.678723 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:40.535020 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:55:40.534988 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:41.411295 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:55:41.411251 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:41.529423 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:55:41.529383 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:41.815215 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:55:41.815181 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:41.949383 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:55:41.949345 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:42.677411 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:55:42.677378 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:43.510441 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:55:43.510413 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:44.258112 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:55:44.258078 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:44.617570 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:55:44.616427 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:44.745545 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:55:44.745513 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:45.192927 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:55:45.192885 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:45.938123 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:55:45.938091 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:47.347121 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:55:47.347090 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:48.376899 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:55:48.376857 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:49.196861 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:55:49.196825 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:50.006649 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:55:50.006619 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:50.943781 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:55:50.943745 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:51.619594 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:55:51.619556 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:52.041312 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:55:52.041226 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:52.716593 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:55:52.716562 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:52.882391 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:55:52.882360 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:53.134595 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:55:53.134557 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:53.263405 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:55:53.263373 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:53.933513 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:55:53.933482 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 18:55:54.225940 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:55:54.225853 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:55.123552 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:55:55.123517 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:56.189887 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:55:56.189853 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:57.287575 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:55:57.287525 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:58.240391 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:55:58.240347 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:59.193669 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:55:59.193635 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:59.329245 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:55:59.329202 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:00.063057 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:56:00.063018 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:00.629102 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:56:00.629073 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:00.846757 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:56:00.846726 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:01.123244 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:56:01.123212 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:01.530991 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:56:01.530890 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:01.821205 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:56:01.821175 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:02.921295 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:56:02.921254 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:03.884768 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:56:03.884729 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:04.683240 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:56:04.683207 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:04.824996 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:56:04.824949 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:05.086921 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:56:05.086889 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:06.186767 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:56:06.186726 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:06.394464 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:56:06.394430 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:07.416104 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:56:07.416074 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:08.822084 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:56:08.822052 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:10.062927 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:56:10.062884 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-2-w2-8a92cc-kserve-6cc4b9c974-n7cd2" podUID="e5fd8f33-8662-4f90-9817-e601f694bf1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:56:10.381344 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:56:10.381242 6960 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-aa6736/simulator-w1-aa6736-kserve-76985dcd5b-njvnn" podUID="943f7840-8d54-4845-b7ec-c0da72ba9c67" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:56:10.562170 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:56:10.561235 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7a8fa2/simulator-w3-7a8fa2-kserve-78587cb8fc-6pvsz" podUID="cc64ebb9-44e9-435a-9db0-d53ce5f139ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:56:10.665224 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:56:10.664235 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7a8fa2/premium-w3-7a8fa2-kserve-6b6c5f6fb8-n5wtq" podUID="f51fd746-cd85-4608-96a6-f54afa0ec34d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:56:10.709171 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:56:10.708225 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-aa6736/premium-w1-aa6736-kserve-6f986b66f7-nkfrg" podUID="84ca40c5-7033-48c4-bcec-e9ac7c034a4e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:56:10.908867 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:56:10.908830 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:11.030245 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:56:11.030164 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:11.365947 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:56:11.365916 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:11.519250 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:56:11.519212 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:12.123552 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:56:12.123522 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:13.043597 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:56:13.043565 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:14.118207 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:56:14.118170 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:15.001368 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:56:15.001330 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:15.115484 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:56:15.115457 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:15.229845 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:56:15.229808 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:16.189293 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:56:16.189257 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:16.992429 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:56:16.992393 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:18.020654 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:56:18.020620 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:18.932630 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:56:18.932595 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:19.915002 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:56:19.914970 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:20.598139 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:56:20.598106 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:20.802529 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:56:20.802496 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:21.014323 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:56:21.008643 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:21.150961 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:56:21.150922 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:21.627528 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:56:21.627494 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:22.077064 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:56:22.077030 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:23.107326 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:56:23.107283 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:23.932430 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:56:23.932397 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 18:56:24.144755 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:56:24.144716 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:24.924016 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:56:24.923983 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:25.880415 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:56:25.880376 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:26.662611 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:56:26.662304 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:26.798539 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:56:26.798508 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:27.642671 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:56:27.642636 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:28.535312 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:56:28.535282 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:29.462802 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:56:29.462745 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:29.959302 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:56:29.959265 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:30.073547 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:56:30.073517 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:30.255276 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:56:30.251944 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:30.511834 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:56:30.511761 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:31.705981 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:56:31.705948 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:32.958169 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:56:32.955314 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:33.966225 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:56:33.964256 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-2-w3-7a8fa2-kserve-677bb44d8d-fg55k" podUID="e8701242-b59c-43b1-9c0a-011635edd419" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 30 18:56:35.166051 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:56:35.165962 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:36.438542 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:56:36.438506 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:37.161584 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:56:37.161550 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:37.274611 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:56:37.274570 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:37.827658 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:56:37.827627 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:38.970660 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:56:38.970626 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:39.847755 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:56:39.847717 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:40.806487 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:56:40.806452 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:41.800171 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:56:41.800123 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:42.655877 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:56:42.655777 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:43.874229 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:56:43.874190 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:44.842449 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:56:44.842407 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:45.912716 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:56:45.912679 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:46.803572 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:56:46.803539 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:47.815135 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:56:47.815097 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:48.692535 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:56:48.692500 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:49.592217 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:56:49.592181 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:50.430117 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:56:50.430050 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:51.368484 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:56:51.368442 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:51.658918 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:56:51.658843 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:52.347116 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:56:52.347082 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:53.103929 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:56:53.103895 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:53.671529 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:56:53.671496 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:53.809830 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:56:53.809789 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:53.991405 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:56:53.991310 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 18:56:54.229808 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:56:54.229774 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:55.365212 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:56:55.365171 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:56.236689 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:56:56.236639 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:57.005948 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:56:57.005913 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:57.310878 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:56:57.310847 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:57.566260 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:56:57.566081 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:57.766172 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:56:57.765745 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:59.197958 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:56:59.197908 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:59.579185 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:56:59.570550 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7a8fa2/simulator-w3-7a8fa2-kserve-78587cb8fc-6pvsz" podUID="cc64ebb9-44e9-435a-9db0-d53ce5f139ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:56:59.687280 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:56:59.687233 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-aa6736/premium-w1-aa6736-kserve-6f986b66f7-nkfrg" podUID="84ca40c5-7033-48c4-bcec-e9ac7c034a4e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 30 18:56:59.736340 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:56:59.736289 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7a8fa2/premium-w3-7a8fa2-kserve-6b6c5f6fb8-n5wtq" podUID="f51fd746-cd85-4608-96a6-f54afa0ec34d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:56:59.766171 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:56:59.759240 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8fq69" podUID="557f8cec-b11d-4faf-8d51-ca780d9381ca" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.26:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:56:59.816171 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:56:59.815235 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/simulator-w2-8a92cc-kserve-fb77d84bc-nqf9q" podUID="ce70492e-c727-4dc1-b2ee-bf278d29a45b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 18:56:59.913770 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:56:59.913676 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-aa6736/distinct-w1-aa6736-kserve-b86d7667b-smcbz" podUID="58c23db7-11cb-4ce2-a67e-fe698e89c71d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:56:59.954280 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:56:59.954224 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-2-w3-7a8fa2-kserve-677bb44d8d-fg55k" podUID="e8701242-b59c-43b1-9c0a-011635edd419" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:56:59.973291 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:56:59.973235 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-aa6736/simulator-w1-aa6736-kserve-76985dcd5b-njvnn" podUID="943f7840-8d54-4845-b7ec-c0da72ba9c67" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:57:00.421973 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:00.421924 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:01.433261 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:01.433213 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:02.320770 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:02.320729 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:03.410363 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:03.410206 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:03.525331 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:03.525288 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:03.870420 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:03.870383 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:04.003296 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:04.003259 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:04.618783 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:04.618753 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:05.093541 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:05.093509 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:05.259451 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:05.259417 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:05.824936 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:05.824897 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:06.321206 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:06.321116 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:06.810626 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:06.810596 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:08.001553 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:08.001375 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:08.879949 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:08.879915 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:09.736968 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:09.736927 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:10.528858 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:10.528824 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:11.371863 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:11.371807 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:11.977002 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:11.976966 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:12.230414 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:12.230337 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:12.340262 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:12.340232 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:12.543261 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:12.543172 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:13.224189 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:13.224156 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:13.444512 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:13.444484 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:13.586675 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:13.586641 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:14.549754 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:14.549723 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:15.604589 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:15.604555 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:16.541775 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:16.541736 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:17.763325 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:17.763289 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:18.915187 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:18.912743 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:19.838989 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:19.838958 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-fa7b20-7ff6f5cfdf-2n8cn"] Oct 30 18:57:19.839451 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:19.839409 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fa7b20-7ff6f5cfdf-2n8cn" podUID="95c445fe-9636-49bd-b910-8ea961f962ce" containerName="maas-api" containerID="cri-o://d6cc0540302d4b26a162a04cd6fded0fe075906fe2a76a6719bfc54109f03854" gracePeriod=30 Oct 30 18:57:20.172226 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:20.170437 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:20.637193 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:20.636715 6960 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-m5wdn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:57:20.637193 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:20.636821 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-m5wdn" podUID="6178626a-4928-40ee-b26a-362f39ee1965" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:57:20.663546 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:20.662513 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-aa6736/premium-w1-aa6736-kserve-6f986b66f7-nkfrg" podUID="84ca40c5-7033-48c4-bcec-e9ac7c034a4e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:57:20.819332 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:20.819288 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/simulator-w2-8a92cc-kserve-fb77d84bc-nqf9q" podUID="ce70492e-c727-4dc1-b2ee-bf278d29a45b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:57:21.231572 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:21.214850 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:21.232266 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:21.232224 6960 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-aa6736/distinct-w1-aa6736-kserve-b86d7667b-smcbz" podUID="58c23db7-11cb-4ce2-a67e-fe698e89c71d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": context deadline exceeded" Oct 30 18:57:21.240171 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:21.232915 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-aa6736/simulator-w1-aa6736-kserve-76985dcd5b-njvnn" podUID="943f7840-8d54-4845-b7ec-c0da72ba9c67" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 30 18:57:21.274409 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:21.240264 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-aa6736/distinct-w1-aa6736-kserve-b86d7667b-smcbz" podUID="58c23db7-11cb-4ce2-a67e-fe698e89c71d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:57:21.296192 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:21.240301 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-2-w3-7a8fa2-kserve-677bb44d8d-fg55k" podUID="e8701242-b59c-43b1-9c0a-011635edd419" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 30 18:57:21.390169 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:21.367862 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:21.403204 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:21.401591 6960 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-7a8fa2/premium-w3-7a8fa2-kserve-6b6c5f6fb8-n5wtq" podUID="f51fd746-cd85-4608-96a6-f54afa0ec34d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:57:21.491226 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:21.487546 6960 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-8a92cc/simulator-w2-8a92cc-kserve-fb77d84bc-nqf9q" podUID="ce70492e-c727-4dc1-b2ee-bf278d29a45b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:57:21.547768 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:21.546576 6960 generic.go:356] "Generic (PLEG): container finished" podID="95c445fe-9636-49bd-b910-8ea961f962ce" containerID="d6cc0540302d4b26a162a04cd6fded0fe075906fe2a76a6719bfc54109f03854" exitCode=0 Oct 30 18:57:21.548463 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:21.546605 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fa7b20-7ff6f5cfdf-2n8cn" event={"ID":"95c445fe-9636-49bd-b910-8ea961f962ce","Type":"ContainerDied","Data":"d6cc0540302d4b26a162a04cd6fded0fe075906fe2a76a6719bfc54109f03854"} Oct 30 18:57:21.583333 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:21.583275 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7a8fa2/simulator-w3-7a8fa2-kserve-78587cb8fc-6pvsz" podUID="cc64ebb9-44e9-435a-9db0-d53ce5f139ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 30 18:57:21.869934 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:21.869899 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:21.897657 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:21.897629 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fa7b20-7ff6f5cfdf-2n8cn" Oct 30 18:57:22.022279 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:22.022251 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95c445fe-9636-49bd-b910-8ea961f962ce-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95c445fe-9636-49bd-b910-8ea961f962ce-maas-api-tls\") pod \"95c445fe-9636-49bd-b910-8ea961f962ce\" (UID: \"95c445fe-9636-49bd-b910-8ea961f962ce\") " Oct 30 18:57:22.022452 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:22.022337 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95c445fe-9636-49bd-b910-8ea961f962ce-kube-api-access-dvkcx\" (UniqueName: \"kubernetes.io/projected/95c445fe-9636-49bd-b910-8ea961f962ce-kube-api-access-dvkcx\") pod \"95c445fe-9636-49bd-b910-8ea961f962ce\" (UID: \"95c445fe-9636-49bd-b910-8ea961f962ce\") " Oct 30 18:57:22.022528 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:22.022448 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95c445fe-9636-49bd-b910-8ea961f962ce-metrics-certs\" (UniqueName: \"kubernetes.io/secret/95c445fe-9636-49bd-b910-8ea961f962ce-metrics-certs\") pod \"95c445fe-9636-49bd-b910-8ea961f962ce\" (UID: \"95c445fe-9636-49bd-b910-8ea961f962ce\") " Oct 30 18:57:22.022657 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:22.022640 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95c445fe-9636-49bd-b910-8ea961f962ce-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95c445fe-9636-49bd-b910-8ea961f962ce-openshift-service-ca\") pod \"95c445fe-9636-49bd-b910-8ea961f962ce\" (UID: \"95c445fe-9636-49bd-b910-8ea961f962ce\") " Oct 30 18:57:22.023329 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:22.023301 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95c445fe-9636-49bd-b910-8ea961f962ce-openshift-service-ca" pod "95c445fe-9636-49bd-b910-8ea961f962ce" (UID: "95c445fe-9636-49bd-b910-8ea961f962ce"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:57:22.023566 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:22.023547 6960 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/95c445fe-9636-49bd-b910-8ea961f962ce-openshift-service-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:57:22.024627 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:22.024563 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95c445fe-9636-49bd-b910-8ea961f962ce-maas-api-tls" pod "95c445fe-9636-49bd-b910-8ea961f962ce" (UID: "95c445fe-9636-49bd-b910-8ea961f962ce"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:57:22.024734 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:22.024629 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95c445fe-9636-49bd-b910-8ea961f962ce-kube-api-access-dvkcx" pod "95c445fe-9636-49bd-b910-8ea961f962ce" (UID: "95c445fe-9636-49bd-b910-8ea961f962ce"). InnerVolumeSpecName "kube-api-access-dvkcx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:57:22.024734 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:22.024670 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95c445fe-9636-49bd-b910-8ea961f962ce-metrics-certs" pod "95c445fe-9636-49bd-b910-8ea961f962ce" (UID: "95c445fe-9636-49bd-b910-8ea961f962ce"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:57:22.033967 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:22.033942 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:22.124657 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:22.124585 6960 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/95c445fe-9636-49bd-b910-8ea961f962ce-maas-api-tls\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:57:22.124657 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:22.124613 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvkcx\" (UniqueName: \"kubernetes.io/projected/95c445fe-9636-49bd-b910-8ea961f962ce-kube-api-access-dvkcx\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:57:22.124657 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:22.124622 6960 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95c445fe-9636-49bd-b910-8ea961f962ce-metrics-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:57:22.556665 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:22.556628 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fa7b20-7ff6f5cfdf-2n8cn" event={"ID":"95c445fe-9636-49bd-b910-8ea961f962ce","Type":"ContainerDied","Data":"53f8c06b3d0a8d3e5a36c922a3d07b3625e8d3a4fcc60e35a2086991fbd72e0c"} Oct 30 18:57:22.556665 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:22.556653 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fa7b20-7ff6f5cfdf-2n8cn" Oct 30 18:57:22.557207 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:22.556696 6960 scope.go:122] "RemoveContainer" containerID="d6cc0540302d4b26a162a04cd6fded0fe075906fe2a76a6719bfc54109f03854" Oct 30 18:57:22.607372 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:22.607339 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-fa7b20-7ff6f5cfdf-2n8cn"] Oct 30 18:57:22.607713 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:22.607692 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-fa7b20-7ff6f5cfdf-2n8cn"] Oct 30 18:57:23.233999 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:23.233963 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:23.933995 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:57:23.933956 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 18:57:24.250733 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:24.250632 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:24.593945 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:24.593911 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95c445fe-9636-49bd-b910-8ea961f962ce" path="/var/lib/kubelet/pods/95c445fe-9636-49bd-b910-8ea961f962ce/volumes" Oct 30 18:57:25.121662 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:25.121632 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:26.240484 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:26.240424 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:26.883263 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:26.883230 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:26.962071 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:26.962041 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-348877-c6895b5c7-4m2kf"] Oct 30 18:57:26.962338 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:26.962314 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-348877-c6895b5c7-4m2kf" podUID="bd7d5d70-3bf1-4b29-947a-0e7e175fabbb" containerName="maas-api" containerID="cri-o://df11d1251fe3fb1eb30520068099bad126a4b8745166f4032ea5147fde8cd8ad" gracePeriod=30 Oct 30 18:57:27.040086 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:27.040052 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:27.169685 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:27.169548 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:27.288305 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:27.288256 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-348877-c6895b5c7-4m2kf" Oct 30 18:57:27.311166 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:27.293016 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:27.427240 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:27.423970 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:27.444166 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:27.432803 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd7d5d70-3bf1-4b29-947a-0e7e175fabbb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd7d5d70-3bf1-4b29-947a-0e7e175fabbb-openshift-service-ca\") pod \"bd7d5d70-3bf1-4b29-947a-0e7e175fabbb\" (UID: \"bd7d5d70-3bf1-4b29-947a-0e7e175fabbb\") " Oct 30 18:57:27.444166 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:27.433000 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd7d5d70-3bf1-4b29-947a-0e7e175fabbb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd7d5d70-3bf1-4b29-947a-0e7e175fabbb-maas-api-tls\") pod \"bd7d5d70-3bf1-4b29-947a-0e7e175fabbb\" (UID: \"bd7d5d70-3bf1-4b29-947a-0e7e175fabbb\") " Oct 30 18:57:27.444166 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:27.433099 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd7d5d70-3bf1-4b29-947a-0e7e175fabbb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd7d5d70-3bf1-4b29-947a-0e7e175fabbb-metrics-certs\") pod \"bd7d5d70-3bf1-4b29-947a-0e7e175fabbb\" (UID: \"bd7d5d70-3bf1-4b29-947a-0e7e175fabbb\") " Oct 30 18:57:27.444166 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:27.442366 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd7d5d70-3bf1-4b29-947a-0e7e175fabbb-kube-api-access-tdzkt\" (UniqueName: \"kubernetes.io/projected/bd7d5d70-3bf1-4b29-947a-0e7e175fabbb-kube-api-access-tdzkt\") pod \"bd7d5d70-3bf1-4b29-947a-0e7e175fabbb\" (UID: \"bd7d5d70-3bf1-4b29-947a-0e7e175fabbb\") " Oct 30 18:57:27.458200 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:27.445211 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd7d5d70-3bf1-4b29-947a-0e7e175fabbb-openshift-service-ca" pod "bd7d5d70-3bf1-4b29-947a-0e7e175fabbb" (UID: "bd7d5d70-3bf1-4b29-947a-0e7e175fabbb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:57:27.464198 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:27.463262 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd7d5d70-3bf1-4b29-947a-0e7e175fabbb-maas-api-tls" pod "bd7d5d70-3bf1-4b29-947a-0e7e175fabbb" (UID: "bd7d5d70-3bf1-4b29-947a-0e7e175fabbb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:57:27.489478 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:27.489436 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd7d5d70-3bf1-4b29-947a-0e7e175fabbb-kube-api-access-tdzkt" pod "bd7d5d70-3bf1-4b29-947a-0e7e175fabbb" (UID: "bd7d5d70-3bf1-4b29-947a-0e7e175fabbb"). InnerVolumeSpecName "kube-api-access-tdzkt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:57:27.489877 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:27.489847 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd7d5d70-3bf1-4b29-947a-0e7e175fabbb-metrics-certs" pod "bd7d5d70-3bf1-4b29-947a-0e7e175fabbb" (UID: "bd7d5d70-3bf1-4b29-947a-0e7e175fabbb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:57:27.544463 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:27.544435 6960 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd7d5d70-3bf1-4b29-947a-0e7e175fabbb-metrics-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:57:27.544597 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:27.544466 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdzkt\" (UniqueName: \"kubernetes.io/projected/bd7d5d70-3bf1-4b29-947a-0e7e175fabbb-kube-api-access-tdzkt\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:57:27.544597 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:27.544482 6960 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd7d5d70-3bf1-4b29-947a-0e7e175fabbb-openshift-service-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:57:27.544597 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:27.544491 6960 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd7d5d70-3bf1-4b29-947a-0e7e175fabbb-maas-api-tls\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:57:27.554524 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:27.554474 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:27.594616 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:27.594578 6960 generic.go:356] "Generic (PLEG): container finished" podID="bd7d5d70-3bf1-4b29-947a-0e7e175fabbb" containerID="df11d1251fe3fb1eb30520068099bad126a4b8745166f4032ea5147fde8cd8ad" exitCode=0 Oct 30 18:57:27.594616 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:27.594618 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-348877-c6895b5c7-4m2kf" Oct 30 18:57:27.594874 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:27.594667 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-348877-c6895b5c7-4m2kf" event={"ID":"bd7d5d70-3bf1-4b29-947a-0e7e175fabbb","Type":"ContainerDied","Data":"df11d1251fe3fb1eb30520068099bad126a4b8745166f4032ea5147fde8cd8ad"} Oct 30 18:57:27.594874 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:27.594726 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-348877-c6895b5c7-4m2kf" event={"ID":"bd7d5d70-3bf1-4b29-947a-0e7e175fabbb","Type":"ContainerDied","Data":"de43292324e24ca08130095410d8a7eace774241db14ee8ab4dfbd5174ee094c"} Oct 30 18:57:27.594874 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:27.594756 6960 scope.go:122] "RemoveContainer" containerID="df11d1251fe3fb1eb30520068099bad126a4b8745166f4032ea5147fde8cd8ad" Oct 30 18:57:27.607363 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:27.607334 6960 scope.go:122] "RemoveContainer" containerID="df11d1251fe3fb1eb30520068099bad126a4b8745166f4032ea5147fde8cd8ad" Oct 30 18:57:27.607800 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:57:27.607739 6960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df11d1251fe3fb1eb30520068099bad126a4b8745166f4032ea5147fde8cd8ad\": container with ID starting with df11d1251fe3fb1eb30520068099bad126a4b8745166f4032ea5147fde8cd8ad not found: ID does not exist" containerID="df11d1251fe3fb1eb30520068099bad126a4b8745166f4032ea5147fde8cd8ad" Oct 30 18:57:27.607901 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:27.607802 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df11d1251fe3fb1eb30520068099bad126a4b8745166f4032ea5147fde8cd8ad"} err="failed to get container status \"df11d1251fe3fb1eb30520068099bad126a4b8745166f4032ea5147fde8cd8ad\": rpc error: code = NotFound desc = could not find container \"df11d1251fe3fb1eb30520068099bad126a4b8745166f4032ea5147fde8cd8ad\": container with ID starting with df11d1251fe3fb1eb30520068099bad126a4b8745166f4032ea5147fde8cd8ad not found: ID does not exist" Oct 30 18:57:27.625694 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:27.625665 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-348877-c6895b5c7-4m2kf"] Oct 30 18:57:27.634176 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:27.634110 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-348877-c6895b5c7-4m2kf"] Oct 30 18:57:27.696707 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:27.696597 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:28.070568 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:28.070525 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:28.604024 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:28.603996 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd7d5d70-3bf1-4b29-947a-0e7e175fabbb" path="/var/lib/kubelet/pods/bd7d5d70-3bf1-4b29-947a-0e7e175fabbb/volumes" Oct 30 18:57:28.634566 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:28.634538 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:29.144106 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:29.144075 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:29.261863 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:29.261834 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:29.517681 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:29.517575 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:29.838377 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:29.838343 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:30.630705 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:30.630670 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:31.640022 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:31.639990 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:32.605387 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:32.605355 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:33.609876 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:33.609839 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:34.957176 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:34.955289 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-2-w3-7a8fa2-kserve-677bb44d8d-fg55k" podUID="e8701242-b59c-43b1-9c0a-011635edd419" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:57:35.001172 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:34.977542 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-aa6736/simulator-w1-aa6736-kserve-76985dcd5b-njvnn" podUID="943f7840-8d54-4845-b7ec-c0da72ba9c67" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 30 18:57:35.075171 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:35.074672 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-2-w2-8a92cc-kserve-6cc4b9c974-n7cd2" podUID="e5fd8f33-8662-4f90-9817-e601f694bf1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:57:35.566297 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:35.566244 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7a8fa2/simulator-w3-7a8fa2-kserve-78587cb8fc-6pvsz" podUID="cc64ebb9-44e9-435a-9db0-d53ce5f139ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:57:35.573179 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:35.568723 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:35.664162 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:35.659617 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-aa6736/premium-w1-aa6736-kserve-6f986b66f7-nkfrg" podUID="84ca40c5-7033-48c4-bcec-e9ac7c034a4e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 30 18:57:35.791724 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:35.791685 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:35.957623 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:35.957532 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:36.216749 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:36.216663 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:36.761945 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:36.761915 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:36.920482 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:36.920450 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:37.624079 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:37.624042 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:37.853178 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:37.853133 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:38.389103 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:38.389041 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:38.949922 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:38.949875 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:39.071392 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:39.071352 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:39.901632 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:39.901598 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:41.022127 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:41.022086 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:41.967726 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:41.967691 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:42.500858 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:42.500816 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:42.625918 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:42.625797 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:43.250964 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:43.250928 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:43.798438 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:43.798405 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:43.952832 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:43.952778 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:44.417803 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:44.416920 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:45.480761 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:45.480728 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:46.344136 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:46.344097 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:47.850071 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:47.849984 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:48.747639 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:48.747600 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:49.602618 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:49.602581 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:50.700386 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:50.700353 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:51.616134 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:51.616095 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:51.733164 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:51.733118 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:51.843926 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:51.843893 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:51.999293 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:51.999207 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:52.729818 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:52.729785 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:53.505163 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:53.505082 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:53.936612 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:57:53.936570 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 18:57:54.332206 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:54.332138 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:55.218022 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:55.217990 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:56.258160 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:56.258112 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:57.860890 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:57.860847 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:59.082435 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:59.082401 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:59.976853 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:57:59.976823 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:01.102676 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:01.102642 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:02.041215 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:02.041179 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:03.181607 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:03.181568 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:04.061580 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:04.061550 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:04.167550 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:04.167517 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:04.921579 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:04.921541 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:05.239134 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:05.239049 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:05.694521 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:05.694484 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:06.527793 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:06.527756 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:07.771422 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:07.771384 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:07.876726 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:07.876687 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:08.670913 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:08.670880 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:09.064026 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:09.063995 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:09.502585 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:09.502500 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:09.658457 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:09.658427 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:09.780929 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:09.780831 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:09.922716 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:09.922679 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:10.101735 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:10.101701 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:10.237950 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:10.237924 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:10.731632 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:10.731599 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:11.583029 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:11.582991 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:13.285169 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:13.284461 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:13.533730 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:13.533696 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:13.675699 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:13.675664 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:13.976636 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:13.976548 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:14.135187 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:14.133747 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:14.437501 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:14.437468 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:14.572172 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:14.569068 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:14.656296 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:14.656254 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-aa6736/premium-w1-aa6736-kserve-6f986b66f7-nkfrg" podUID="84ca40c5-7033-48c4-bcec-e9ac7c034a4e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 30 18:58:14.962524 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:14.962480 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-2-w3-7a8fa2-kserve-677bb44d8d-fg55k" podUID="e8701242-b59c-43b1-9c0a-011635edd419" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:58:14.982413 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:14.982366 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-aa6736/simulator-w1-aa6736-kserve-76985dcd5b-njvnn" podUID="943f7840-8d54-4845-b7ec-c0da72ba9c67" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded" Oct 30 18:58:15.085169 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:15.078966 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-2-w2-8a92cc-kserve-6cc4b9c974-n7cd2" podUID="e5fd8f33-8662-4f90-9817-e601f694bf1d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:58:16.168636 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:16.168602 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:16.578222 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:16.578134 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:16.694518 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:16.694479 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:17.267482 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:17.267395 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:18.207624 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:18.207584 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:19.009848 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:19.009794 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:19.158998 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:19.158964 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:19.284825 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:19.284736 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:19.443467 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:19.443437 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:20.494677 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:20.494644 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:21.630377 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:21.629756 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:22.467642 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:22.467612 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:23.348586 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:23.348559 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:23.934039 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:58:23.934003 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 18:58:24.178618 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:24.178588 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:24.298687 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:24.298562 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:24.414325 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:24.414286 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:25.079618 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:25.079585 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:25.260330 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:25.260294 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:25.556777 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:25.556735 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:25.673666 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:25.673634 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:26.433258 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:26.433218 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:27.243998 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:27.243961 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:28.133640 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:28.133604 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:29.377542 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:29.377510 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:29.565630 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:29.565593 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:29.740812 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:29.740725 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:29.930652 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:29.930620 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:30.691123 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:30.691091 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:30.828559 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:30.828523 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:30.992225 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:30.988098 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:31.738779 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:31.738738 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:32.520996 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:32.520960 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:33.495067 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:33.495031 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:34.299762 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:34.299723 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:35.203802 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:35.203771 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:36.095701 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:36.095663 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:37.010066 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:37.010028 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:38.300459 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:38.300422 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:38.438891 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:38.438856 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:39.080563 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:39.080527 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:39.347228 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:39.337519 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:39.716566 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:39.715871 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:39.869489 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:39.869449 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:40.359253 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:40.359216 6960 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-aa6736/simulator-w1-aa6736-kserve-76985dcd5b-njvnn" podUID="943f7840-8d54-4845-b7ec-c0da72ba9c67" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": context deadline exceeded" Oct 30 18:58:40.382631 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:40.382599 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:40.567594 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:40.566308 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7a8fa2/simulator-w3-7a8fa2-kserve-78587cb8fc-6pvsz" podUID="cc64ebb9-44e9-435a-9db0-d53ce5f139ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:58:40.602349 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:40.602268 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:40.666347 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:40.666242 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-aa6736/premium-w1-aa6736-kserve-6f986b66f7-nkfrg" podUID="84ca40c5-7033-48c4-bcec-e9ac7c034a4e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:58:40.666974 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:40.666941 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7a8fa2/premium-w3-7a8fa2-kserve-6b6c5f6fb8-n5wtq" podUID="f51fd746-cd85-4608-96a6-f54afa0ec34d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 30 18:58:40.887302 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:40.887245 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/simulator-w2-8a92cc-kserve-fb77d84bc-nqf9q" podUID="ce70492e-c727-4dc1-b2ee-bf278d29a45b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:58:40.938223 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:40.931220 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-aa6736/distinct-w1-aa6736-kserve-b86d7667b-smcbz" podUID="58c23db7-11cb-4ce2-a67e-fe698e89c71d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 18:58:40.969272 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:40.963892 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:40.988215 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:40.986619 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-aa6736/simulator-w1-aa6736-kserve-76985dcd5b-njvnn" podUID="943f7840-8d54-4845-b7ec-c0da72ba9c67" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:58:40.988550 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:40.988523 6960 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-7a8fa2/premium-w3-7a8fa2-kserve-6b6c5f6fb8-n5wtq" podUID="f51fd746-cd85-4608-96a6-f54afa0ec34d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:58:41.044562 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:41.044505 6960 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-aa6736/distinct-w1-aa6736-kserve-b86d7667b-smcbz" podUID="58c23db7-11cb-4ce2-a67e-fe698e89c71d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:58:41.477906 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:41.477868 6960 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-8a92cc/simulator-w2-8a92cc-kserve-fb77d84bc-nqf9q" podUID="ce70492e-c727-4dc1-b2ee-bf278d29a45b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": context deadline exceeded" Oct 30 18:58:41.524970 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:41.524621 6960 patch_prober.go:29] interesting pod/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-5m6bs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:58:41.524970 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:41.524720 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-5m6bs" podUID="f0a610f7-6daf-48de-865d-a0788c9d036a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:58:41.574089 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:41.574056 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:41.722426 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:41.722283 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:42.683297 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:42.683253 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:42.932462 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:42.932421 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:43.069033 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:43.068999 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:43.760800 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:43.760765 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:44.170357 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:44.170302 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:44.715496 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:44.715454 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:44.832594 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:44.832559 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:44.981983 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:44.981903 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:45.738509 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:45.738478 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:46.659279 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:46.659238 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:47.760291 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:47.760260 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:48.418940 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:48.418901 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:48.621369 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:48.621332 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:48.867051 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:48.867003 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:49.024720 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:49.024687 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:49.248912 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:49.248833 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:49.793296 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:49.793264 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:50.019671 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:50.019639 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:50.379506 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:50.379469 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:50.533920 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:50.533891 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:50.748191 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:50.748089 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:51.089471 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:51.089360 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:52.131164 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:52.124535 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:52.321506 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:52.321467 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:53.384106 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:53.384066 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:53.934380 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:58:53.934344 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 18:58:54.303336 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:54.303248 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:54.622849 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:54.622807 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:55.183937 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:55.183901 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:55.365130 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:55.365083 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:55.599482 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:55.599448 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:55.732781 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:55.732744 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:55.853087 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:55.853006 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:56.212041 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:56.211945 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:56.360935 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:56.360899 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:56.819599 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:56.819561 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:57.392116 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:57.392075 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:58.213781 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:58.213744 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:58.316555 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:58.316521 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:58.533179 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:58.533081 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:58.659247 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:58.659212 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:59.477054 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:59.477018 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:59.636172 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:59.630113 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:59.804787 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:58:59.804757 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:00.060625 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:00.060529 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:00.869551 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:00.869521 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:01.036092 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:01.036056 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:01.677785 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:01.677757 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:01.825564 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:01.825444 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:02.181891 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:02.181859 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:03.317026 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:03.316992 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:04.309491 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:04.309457 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:05.377532 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:05.377499 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:06.123886 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:06.123849 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:06.386219 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:06.382160 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:06.658601 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:06.658511 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:06.766943 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:06.766897 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:07.222641 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:07.222610 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:07.757216 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:07.757184 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:09.641063 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:09.641019 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7a8fa2/simulator-w3-7a8fa2-kserve-78587cb8fc-6pvsz" podUID="cc64ebb9-44e9-435a-9db0-d53ce5f139ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:59:09.655267 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:09.655235 6960 patch_prober.go:29] interesting pod/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ng5vb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:59:09.655427 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:09.655326 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ng5vb" podUID="f1ef57ac-d33d-4998-b22b-87b2aed6ba8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:59:09.915224 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:09.914164 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:10.965384 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:10.965351 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:11.069073 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:11.069035 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:11.194883 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:11.194821 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:11.627230 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:11.627198 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-4thf5"] Oct 30 18:59:11.627695 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:11.627663 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-4thf5" podUID="1c4b2f62-76f9-466c-b747-ee608a8bc6e3" containerName="istio-proxy" containerID="cri-o://94a79476ef6b20859e2524b21ad96a7da3bfee23909321b8534ebbce294aacc2" gracePeriod=30 Oct 30 18:59:11.635646 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:11.635621 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-c2m5d"] Oct 30 18:59:11.636026 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:11.635997 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-c2m5d" podUID="090b2a4e-323a-4a3c-9a5b-165df26c5b56" containerName="istio-proxy" containerID="cri-o://a50a647368a401a5e583a95ab90cb812dfe50d2f09364f5ef255fdefa4345aa8" gracePeriod=30 Oct 30 18:59:12.109618 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:12.109565 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:12.311864 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:12.311831 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:13.167416 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:13.167372 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:13.661292 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:13.661258 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:14.102978 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:14.102941 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:14.215443 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:14.215406 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:14.688556 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:14.672225 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:14.795164 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:14.794803 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:15.147025 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:15.146992 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:15.284413 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:15.284191 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:15.414867 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:15.414776 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:15.543869 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:15.543839 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:15.677222 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:15.676559 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:15.821751 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:15.821718 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:16.277354 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:16.277323 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:16.546225 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:16.546119 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:16.668942 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:16.668904 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:16.954913 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:16.954783 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-c2m5d" Oct 30 18:59:16.974912 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:16.974890 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-4thf5" Oct 30 18:59:17.096808 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:17.096720 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c4b2f62-76f9-466c-b747-ee608a8bc6e3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c4b2f62-76f9-466c-b747-ee608a8bc6e3-istiod-ca-cert\") pod \"1c4b2f62-76f9-466c-b747-ee608a8bc6e3\" (UID: \"1c4b2f62-76f9-466c-b747-ee608a8bc6e3\") " Oct 30 18:59:17.096808 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:17.096784 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1c4b2f62-76f9-466c-b747-ee608a8bc6e3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c4b2f62-76f9-466c-b747-ee608a8bc6e3-istio-podinfo\") pod \"1c4b2f62-76f9-466c-b747-ee608a8bc6e3\" (UID: \"1c4b2f62-76f9-466c-b747-ee608a8bc6e3\") " Oct 30 18:59:17.097056 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:17.096823 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c4b2f62-76f9-466c-b747-ee608a8bc6e3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c4b2f62-76f9-466c-b747-ee608a8bc6e3-istio-data\") pod \"1c4b2f62-76f9-466c-b747-ee608a8bc6e3\" (UID: \"1c4b2f62-76f9-466c-b747-ee608a8bc6e3\") " Oct 30 18:59:17.097056 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:17.096867 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/090b2a4e-323a-4a3c-9a5b-165df26c5b56-kube-api-access-tsj5n\" (UniqueName: \"kubernetes.io/projected/090b2a4e-323a-4a3c-9a5b-165df26c5b56-kube-api-access-tsj5n\") pod \"090b2a4e-323a-4a3c-9a5b-165df26c5b56\" (UID: \"090b2a4e-323a-4a3c-9a5b-165df26c5b56\") " Oct 30 18:59:17.097056 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:17.096939 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/090b2a4e-323a-4a3c-9a5b-165df26c5b56-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/090b2a4e-323a-4a3c-9a5b-165df26c5b56-istiod-ca-cert\") pod \"090b2a4e-323a-4a3c-9a5b-165df26c5b56\" (UID: \"090b2a4e-323a-4a3c-9a5b-165df26c5b56\") " Oct 30 18:59:17.097056 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:17.096996 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c4b2f62-76f9-466c-b747-ee608a8bc6e3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c4b2f62-76f9-466c-b747-ee608a8bc6e3-workload-socket\") pod \"1c4b2f62-76f9-466c-b747-ee608a8bc6e3\" (UID: \"1c4b2f62-76f9-466c-b747-ee608a8bc6e3\") " Oct 30 18:59:17.097056 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:17.097029 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c4b2f62-76f9-466c-b747-ee608a8bc6e3-istio-token\" (UniqueName: \"kubernetes.io/projected/1c4b2f62-76f9-466c-b747-ee608a8bc6e3-istio-token\") pod \"1c4b2f62-76f9-466c-b747-ee608a8bc6e3\" (UID: \"1c4b2f62-76f9-466c-b747-ee608a8bc6e3\") " Oct 30 18:59:17.097368 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:17.097068 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/090b2a4e-323a-4a3c-9a5b-165df26c5b56-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/090b2a4e-323a-4a3c-9a5b-165df26c5b56-istio-data\") pod \"090b2a4e-323a-4a3c-9a5b-165df26c5b56\" (UID: \"090b2a4e-323a-4a3c-9a5b-165df26c5b56\") " Oct 30 18:59:17.097368 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:17.097105 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c4b2f62-76f9-466c-b747-ee608a8bc6e3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c4b2f62-76f9-466c-b747-ee608a8bc6e3-istio-envoy\") pod \"1c4b2f62-76f9-466c-b747-ee608a8bc6e3\" (UID: \"1c4b2f62-76f9-466c-b747-ee608a8bc6e3\") " Oct 30 18:59:17.097368 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:17.097131 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c4b2f62-76f9-466c-b747-ee608a8bc6e3-istiod-ca-cert" pod "1c4b2f62-76f9-466c-b747-ee608a8bc6e3" (UID: "1c4b2f62-76f9-466c-b747-ee608a8bc6e3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:59:17.097368 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:17.097167 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c4b2f62-76f9-466c-b747-ee608a8bc6e3-kube-api-access-nt4dm\" (UniqueName: \"kubernetes.io/projected/1c4b2f62-76f9-466c-b747-ee608a8bc6e3-kube-api-access-nt4dm\") pod \"1c4b2f62-76f9-466c-b747-ee608a8bc6e3\" (UID: \"1c4b2f62-76f9-466c-b747-ee608a8bc6e3\") " Oct 30 18:59:17.097368 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:17.097220 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/090b2a4e-323a-4a3c-9a5b-165df26c5b56-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/090b2a4e-323a-4a3c-9a5b-165df26c5b56-istio-podinfo\") pod \"090b2a4e-323a-4a3c-9a5b-165df26c5b56\" (UID: \"090b2a4e-323a-4a3c-9a5b-165df26c5b56\") " Oct 30 18:59:17.097368 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:17.097262 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/090b2a4e-323a-4a3c-9a5b-165df26c5b56-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/090b2a4e-323a-4a3c-9a5b-165df26c5b56-istio-envoy\") pod \"090b2a4e-323a-4a3c-9a5b-165df26c5b56\" (UID: \"090b2a4e-323a-4a3c-9a5b-165df26c5b56\") " Oct 30 18:59:17.097368 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:17.097327 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/090b2a4e-323a-4a3c-9a5b-165df26c5b56-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/090b2a4e-323a-4a3c-9a5b-165df26c5b56-workload-certs\") pod \"090b2a4e-323a-4a3c-9a5b-165df26c5b56\" (UID: \"090b2a4e-323a-4a3c-9a5b-165df26c5b56\") " Oct 30 18:59:17.097717 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:17.097357 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c4b2f62-76f9-466c-b747-ee608a8bc6e3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c4b2f62-76f9-466c-b747-ee608a8bc6e3-credential-socket\") pod \"1c4b2f62-76f9-466c-b747-ee608a8bc6e3\" (UID: \"1c4b2f62-76f9-466c-b747-ee608a8bc6e3\") " Oct 30 18:59:17.097717 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:17.097417 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/090b2a4e-323a-4a3c-9a5b-165df26c5b56-istio-token\" (UniqueName: \"kubernetes.io/projected/090b2a4e-323a-4a3c-9a5b-165df26c5b56-istio-token\") pod \"090b2a4e-323a-4a3c-9a5b-165df26c5b56\" (UID: \"090b2a4e-323a-4a3c-9a5b-165df26c5b56\") " Oct 30 18:59:17.097717 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:17.097417 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/090b2a4e-323a-4a3c-9a5b-165df26c5b56-istio-data" pod "090b2a4e-323a-4a3c-9a5b-165df26c5b56" (UID: "090b2a4e-323a-4a3c-9a5b-165df26c5b56"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:59:17.097717 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:17.097487 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/090b2a4e-323a-4a3c-9a5b-165df26c5b56-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/090b2a4e-323a-4a3c-9a5b-165df26c5b56-workload-socket\") pod \"090b2a4e-323a-4a3c-9a5b-165df26c5b56\" (UID: \"090b2a4e-323a-4a3c-9a5b-165df26c5b56\") " Oct 30 18:59:17.097717 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:17.097526 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/090b2a4e-323a-4a3c-9a5b-165df26c5b56-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/090b2a4e-323a-4a3c-9a5b-165df26c5b56-credential-socket\") pod \"090b2a4e-323a-4a3c-9a5b-165df26c5b56\" (UID: \"090b2a4e-323a-4a3c-9a5b-165df26c5b56\") " Oct 30 18:59:17.097717 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:17.097622 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c4b2f62-76f9-466c-b747-ee608a8bc6e3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c4b2f62-76f9-466c-b747-ee608a8bc6e3-workload-certs\") pod \"1c4b2f62-76f9-466c-b747-ee608a8bc6e3\" (UID: \"1c4b2f62-76f9-466c-b747-ee608a8bc6e3\") " Oct 30 18:59:17.098039 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:17.097775 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c4b2f62-76f9-466c-b747-ee608a8bc6e3-workload-socket" pod "1c4b2f62-76f9-466c-b747-ee608a8bc6e3" (UID: "1c4b2f62-76f9-466c-b747-ee608a8bc6e3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:59:17.098097 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:17.098065 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/090b2a4e-323a-4a3c-9a5b-165df26c5b56-istiod-ca-cert" pod "090b2a4e-323a-4a3c-9a5b-165df26c5b56" (UID: "090b2a4e-323a-4a3c-9a5b-165df26c5b56"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:59:17.098318 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:17.098294 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c4b2f62-76f9-466c-b747-ee608a8bc6e3-workload-certs" pod "1c4b2f62-76f9-466c-b747-ee608a8bc6e3" (UID: "1c4b2f62-76f9-466c-b747-ee608a8bc6e3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:59:17.098422 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:17.098394 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/090b2a4e-323a-4a3c-9a5b-165df26c5b56-workload-certs" pod "090b2a4e-323a-4a3c-9a5b-165df26c5b56" (UID: "090b2a4e-323a-4a3c-9a5b-165df26c5b56"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:59:17.098553 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:17.098535 6960 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/090b2a4e-323a-4a3c-9a5b-165df26c5b56-workload-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:59:17.098718 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:17.098561 6960 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c4b2f62-76f9-466c-b747-ee608a8bc6e3-workload-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:59:17.098718 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:17.098577 6960 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c4b2f62-76f9-466c-b747-ee608a8bc6e3-istiod-ca-cert\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:59:17.098718 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:17.098578 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c4b2f62-76f9-466c-b747-ee608a8bc6e3-credential-socket" pod "1c4b2f62-76f9-466c-b747-ee608a8bc6e3" (UID: "1c4b2f62-76f9-466c-b747-ee608a8bc6e3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:59:17.098718 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:17.098592 6960 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/090b2a4e-323a-4a3c-9a5b-165df26c5b56-istiod-ca-cert\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:59:17.098718 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:17.098608 6960 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c4b2f62-76f9-466c-b747-ee608a8bc6e3-workload-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:59:17.098718 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:17.098622 6960 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/090b2a4e-323a-4a3c-9a5b-165df26c5b56-istio-data\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:59:17.098718 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:17.098624 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c4b2f62-76f9-466c-b747-ee608a8bc6e3-istio-data" pod "1c4b2f62-76f9-466c-b747-ee608a8bc6e3" (UID: "1c4b2f62-76f9-466c-b747-ee608a8bc6e3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:59:17.099202 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:17.099172 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/090b2a4e-323a-4a3c-9a5b-165df26c5b56-workload-socket" pod "090b2a4e-323a-4a3c-9a5b-165df26c5b56" (UID: "090b2a4e-323a-4a3c-9a5b-165df26c5b56"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:59:17.099316 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:17.099297 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/090b2a4e-323a-4a3c-9a5b-165df26c5b56-credential-socket" pod "090b2a4e-323a-4a3c-9a5b-165df26c5b56" (UID: "090b2a4e-323a-4a3c-9a5b-165df26c5b56"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:59:17.101476 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:17.101436 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/090b2a4e-323a-4a3c-9a5b-165df26c5b56-istio-token" pod "090b2a4e-323a-4a3c-9a5b-165df26c5b56" (UID: "090b2a4e-323a-4a3c-9a5b-165df26c5b56"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:59:17.101592 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:17.101555 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c4b2f62-76f9-466c-b747-ee608a8bc6e3-istio-token" pod "1c4b2f62-76f9-466c-b747-ee608a8bc6e3" (UID: "1c4b2f62-76f9-466c-b747-ee608a8bc6e3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:59:17.102012 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:17.101980 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/090b2a4e-323a-4a3c-9a5b-165df26c5b56-istio-podinfo" pod "090b2a4e-323a-4a3c-9a5b-165df26c5b56" (UID: "090b2a4e-323a-4a3c-9a5b-165df26c5b56"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:59:17.102489 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:17.102461 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c4b2f62-76f9-466c-b747-ee608a8bc6e3-istio-envoy" pod "1c4b2f62-76f9-466c-b747-ee608a8bc6e3" (UID: "1c4b2f62-76f9-466c-b747-ee608a8bc6e3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:59:17.103975 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:17.103947 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c4b2f62-76f9-466c-b747-ee608a8bc6e3-kube-api-access-nt4dm" pod "1c4b2f62-76f9-466c-b747-ee608a8bc6e3" (UID: "1c4b2f62-76f9-466c-b747-ee608a8bc6e3"). InnerVolumeSpecName "kube-api-access-nt4dm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:59:17.104098 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:17.104012 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/090b2a4e-323a-4a3c-9a5b-165df26c5b56-kube-api-access-tsj5n" pod "090b2a4e-323a-4a3c-9a5b-165df26c5b56" (UID: "090b2a4e-323a-4a3c-9a5b-165df26c5b56"). InnerVolumeSpecName "kube-api-access-tsj5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:59:17.104098 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:17.104023 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/090b2a4e-323a-4a3c-9a5b-165df26c5b56-istio-envoy" pod "090b2a4e-323a-4a3c-9a5b-165df26c5b56" (UID: "090b2a4e-323a-4a3c-9a5b-165df26c5b56"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:59:17.104260 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:17.104131 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1c4b2f62-76f9-466c-b747-ee608a8bc6e3-istio-podinfo" pod "1c4b2f62-76f9-466c-b747-ee608a8bc6e3" (UID: "1c4b2f62-76f9-466c-b747-ee608a8bc6e3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:59:17.200529 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:17.200497 6960 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c4b2f62-76f9-466c-b747-ee608a8bc6e3-istio-token\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:59:17.200529 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:17.200526 6960 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c4b2f62-76f9-466c-b747-ee608a8bc6e3-istio-envoy\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:59:17.200779 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:17.200542 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nt4dm\" (UniqueName: \"kubernetes.io/projected/1c4b2f62-76f9-466c-b747-ee608a8bc6e3-kube-api-access-nt4dm\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:59:17.200779 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:17.200556 6960 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/090b2a4e-323a-4a3c-9a5b-165df26c5b56-istio-podinfo\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:59:17.200779 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:17.200568 6960 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/090b2a4e-323a-4a3c-9a5b-165df26c5b56-istio-envoy\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:59:17.200779 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:17.200579 6960 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c4b2f62-76f9-466c-b747-ee608a8bc6e3-credential-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:59:17.200779 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:17.200591 6960 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/090b2a4e-323a-4a3c-9a5b-165df26c5b56-istio-token\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:59:17.200779 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:17.200603 6960 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/090b2a4e-323a-4a3c-9a5b-165df26c5b56-workload-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:59:17.200779 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:17.200615 6960 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/090b2a4e-323a-4a3c-9a5b-165df26c5b56-credential-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:59:17.200779 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:17.200626 6960 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c4b2f62-76f9-466c-b747-ee608a8bc6e3-istio-podinfo\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:59:17.200779 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:17.200638 6960 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c4b2f62-76f9-466c-b747-ee608a8bc6e3-istio-data\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:59:17.200779 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:17.200649 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsj5n\" (UniqueName: \"kubernetes.io/projected/090b2a4e-323a-4a3c-9a5b-165df26c5b56-kube-api-access-tsj5n\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:59:17.309703 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:17.309666 6960 generic.go:356] "Generic (PLEG): container finished" podID="1c4b2f62-76f9-466c-b747-ee608a8bc6e3" containerID="94a79476ef6b20859e2524b21ad96a7da3bfee23909321b8534ebbce294aacc2" exitCode=0 Oct 30 18:59:17.309906 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:17.309728 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-4thf5" Oct 30 18:59:17.309906 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:17.309774 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-4thf5" event={"ID":"1c4b2f62-76f9-466c-b747-ee608a8bc6e3","Type":"ContainerDied","Data":"94a79476ef6b20859e2524b21ad96a7da3bfee23909321b8534ebbce294aacc2"} Oct 30 18:59:17.309906 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:17.309849 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-4thf5" event={"ID":"1c4b2f62-76f9-466c-b747-ee608a8bc6e3","Type":"ContainerDied","Data":"cba0185495e59ce0477adc09bdd476b5980bede1b2bb24e6a114be902b3024e0"} Oct 30 18:59:17.309906 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:17.309879 6960 scope.go:122] "RemoveContainer" containerID="94a79476ef6b20859e2524b21ad96a7da3bfee23909321b8534ebbce294aacc2" Oct 30 18:59:17.312634 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:17.312609 6960 generic.go:356] "Generic (PLEG): container finished" podID="090b2a4e-323a-4a3c-9a5b-165df26c5b56" containerID="a50a647368a401a5e583a95ab90cb812dfe50d2f09364f5ef255fdefa4345aa8" exitCode=0 Oct 30 18:59:17.312763 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:17.312671 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-c2m5d" Oct 30 18:59:17.312763 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:17.312687 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-c2m5d" event={"ID":"090b2a4e-323a-4a3c-9a5b-165df26c5b56","Type":"ContainerDied","Data":"a50a647368a401a5e583a95ab90cb812dfe50d2f09364f5ef255fdefa4345aa8"} Oct 30 18:59:17.312763 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:17.312728 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-c2m5d" event={"ID":"090b2a4e-323a-4a3c-9a5b-165df26c5b56","Type":"ContainerDied","Data":"c720afc5ba5ef50998fbc92999eab4b531664b064f98f3aa9e4dec67e9d235ad"} Oct 30 18:59:17.323084 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:17.323066 6960 scope.go:122] "RemoveContainer" containerID="94a79476ef6b20859e2524b21ad96a7da3bfee23909321b8534ebbce294aacc2" Oct 30 18:59:17.323436 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:59:17.323411 6960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94a79476ef6b20859e2524b21ad96a7da3bfee23909321b8534ebbce294aacc2\": container with ID starting with 94a79476ef6b20859e2524b21ad96a7da3bfee23909321b8534ebbce294aacc2 not found: ID does not exist" containerID="94a79476ef6b20859e2524b21ad96a7da3bfee23909321b8534ebbce294aacc2" Oct 30 18:59:17.323536 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:17.323447 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94a79476ef6b20859e2524b21ad96a7da3bfee23909321b8534ebbce294aacc2"} err="failed to get container status \"94a79476ef6b20859e2524b21ad96a7da3bfee23909321b8534ebbce294aacc2\": rpc error: code = NotFound desc = could not find container \"94a79476ef6b20859e2524b21ad96a7da3bfee23909321b8534ebbce294aacc2\": container with ID starting with 94a79476ef6b20859e2524b21ad96a7da3bfee23909321b8534ebbce294aacc2 not found: ID does not exist" Oct 30 18:59:17.323536 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:17.323473 6960 scope.go:122] "RemoveContainer" containerID="a50a647368a401a5e583a95ab90cb812dfe50d2f09364f5ef255fdefa4345aa8" Oct 30 18:59:17.333966 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:17.333951 6960 scope.go:122] "RemoveContainer" containerID="a50a647368a401a5e583a95ab90cb812dfe50d2f09364f5ef255fdefa4345aa8" Oct 30 18:59:17.334396 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:59:17.334368 6960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a50a647368a401a5e583a95ab90cb812dfe50d2f09364f5ef255fdefa4345aa8\": container with ID starting with a50a647368a401a5e583a95ab90cb812dfe50d2f09364f5ef255fdefa4345aa8 not found: ID does not exist" containerID="a50a647368a401a5e583a95ab90cb812dfe50d2f09364f5ef255fdefa4345aa8" Oct 30 18:59:17.334496 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:17.334404 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a50a647368a401a5e583a95ab90cb812dfe50d2f09364f5ef255fdefa4345aa8"} err="failed to get container status \"a50a647368a401a5e583a95ab90cb812dfe50d2f09364f5ef255fdefa4345aa8\": rpc error: code = NotFound desc = could not find container \"a50a647368a401a5e583a95ab90cb812dfe50d2f09364f5ef255fdefa4345aa8\": container with ID starting with a50a647368a401a5e583a95ab90cb812dfe50d2f09364f5ef255fdefa4345aa8 not found: ID does not exist" Oct 30 18:59:17.340205 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:17.340185 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-4thf5"] Oct 30 18:59:17.346306 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:17.346286 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-4thf5"] Oct 30 18:59:17.366060 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:17.365997 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-c2m5d"] Oct 30 18:59:17.373068 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:17.373048 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-c2m5d"] Oct 30 18:59:17.738170 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:17.738121 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:18.025003 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:18.024882 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:18.146321 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:18.146275 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:18.598192 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:18.598131 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="090b2a4e-323a-4a3c-9a5b-165df26c5b56" path="/var/lib/kubelet/pods/090b2a4e-323a-4a3c-9a5b-165df26c5b56/volumes" Oct 30 18:59:18.598850 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:18.598820 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c4b2f62-76f9-466c-b747-ee608a8bc6e3" path="/var/lib/kubelet/pods/1c4b2f62-76f9-466c-b747-ee608a8bc6e3/volumes" Oct 30 18:59:18.692100 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:18.692055 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-vjb5b"] Oct 30 18:59:18.692341 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:18.692319 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-vjb5b" podUID="160390da-b35c-4030-9fff-d93afbcf9bad" containerName="istio-proxy" containerID="cri-o://af04992b96a297bcf36ebea4a331cfe69c8e368f7ccc7012f4e8851315113726" gracePeriod=30 Oct 30 18:59:18.782386 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:18.782346 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-n9mh9"] Oct 30 18:59:18.782769 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:18.782624 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-n9mh9" podUID="6eaab09d-e908-4663-9793-30cff7591dfb" containerName="istio-proxy" containerID="cri-o://efb91e4d840cdee5f09dd66ad3266a4675fbf390be43ebefd7e21c85c9fca6bd" gracePeriod=30 Oct 30 18:59:18.969641 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:18.969543 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:19.345340 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:19.345303 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:19.400546 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:19.400518 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-03362d-5df77d4775-tn9s2"] Oct 30 18:59:19.401115 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:19.401099 6960 cpu_manager.go:414] "containerMap: removing container" podUID="1c4b2f62-76f9-466c-b747-ee608a8bc6e3" containerName="istio-proxy" Oct 30 18:59:19.401188 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:19.401117 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c4b2f62-76f9-466c-b747-ee608a8bc6e3" containerName="istio-proxy" Oct 30 18:59:19.401188 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:19.401158 6960 cpu_manager.go:414] "containerMap: removing container" podUID="95c445fe-9636-49bd-b910-8ea961f962ce" containerName="maas-api" Oct 30 18:59:19.401188 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:19.401166 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95c445fe-9636-49bd-b910-8ea961f962ce" containerName="maas-api" Oct 30 18:59:19.401188 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:19.401177 6960 cpu_manager.go:414] "containerMap: removing container" podUID="090b2a4e-323a-4a3c-9a5b-165df26c5b56" containerName="istio-proxy" Oct 30 18:59:19.401188 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:19.401181 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="090b2a4e-323a-4a3c-9a5b-165df26c5b56" containerName="istio-proxy" Oct 30 18:59:19.401362 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:19.401196 6960 cpu_manager.go:414] "containerMap: removing container" podUID="bd7d5d70-3bf1-4b29-947a-0e7e175fabbb" containerName="maas-api" Oct 30 18:59:19.401362 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:19.401201 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd7d5d70-3bf1-4b29-947a-0e7e175fabbb" containerName="maas-api" Oct 30 18:59:19.401362 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:19.401304 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="95c445fe-9636-49bd-b910-8ea961f962ce" containerName="maas-api" Oct 30 18:59:19.401362 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:19.401317 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="090b2a4e-323a-4a3c-9a5b-165df26c5b56" containerName="istio-proxy" Oct 30 18:59:19.401362 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:19.401325 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd7d5d70-3bf1-4b29-947a-0e7e175fabbb" containerName="maas-api" Oct 30 18:59:19.401362 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:19.401337 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c4b2f62-76f9-466c-b747-ee608a8bc6e3" containerName="istio-proxy" Oct 30 18:59:19.406126 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:19.406109 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-03362d-5df77d4775-tn9s2" Oct 30 18:59:19.415195 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:19.415175 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-03362d-dockercfg-vcb96\"" Oct 30 18:59:19.422242 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:19.422190 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-03362d\"" Oct 30 18:59:19.423638 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:19.423616 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-03362d-5df77d4775-tn9s2"] Oct 30 18:59:19.454467 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:19.454435 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-03362d-655b6cd7f8-ndl7x"] Oct 30 18:59:19.461351 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:19.461324 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-03362d-655b6cd7f8-ndl7x" Oct 30 18:59:19.469539 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:19.469516 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-03362d-655b6cd7f8-ndl7x"] Oct 30 18:59:19.487777 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:19.487732 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:19.543254 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:19.543217 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpc7q\" (UniqueName: \"kubernetes.io/projected/c8690e67-62d5-4a1b-8632-a23a7d2f2843-kube-api-access-qpc7q\") pod \"payload-pre-processing-e2e-shared-b-w4-03362d-5df77d4775-tn9s2\" (UID: \"c8690e67-62d5-4a1b-8632-a23a7d2f2843\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-03362d-5df77d4775-tn9s2" Oct 30 18:59:19.543441 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:19.543293 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8690e67-62d5-4a1b-8632-a23a7d2f2843-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-03362d-5df77d4775-tn9s2\" (UID: \"c8690e67-62d5-4a1b-8632-a23a7d2f2843\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-03362d-5df77d4775-tn9s2" Oct 30 18:59:19.543441 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:19.543319 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5a06a95-d251-46b6-889c-7e7d5e2ca925-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-03362d-655b6cd7f8-ndl7x\" (UID: \"b5a06a95-d251-46b6-889c-7e7d5e2ca925\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-03362d-655b6cd7f8-ndl7x" Oct 30 18:59:19.543565 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:19.543535 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9rwr\" (UniqueName: \"kubernetes.io/projected/b5a06a95-d251-46b6-889c-7e7d5e2ca925-kube-api-access-b9rwr\") pod \"payload-processing-e2e-shared-b-w4-03362d-655b6cd7f8-ndl7x\" (UID: \"b5a06a95-d251-46b6-889c-7e7d5e2ca925\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-03362d-655b6cd7f8-ndl7x" Oct 30 18:59:19.644526 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:19.644444 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpc7q\" (UniqueName: \"kubernetes.io/projected/c8690e67-62d5-4a1b-8632-a23a7d2f2843-kube-api-access-qpc7q\") pod \"payload-pre-processing-e2e-shared-b-w4-03362d-5df77d4775-tn9s2\" (UID: \"c8690e67-62d5-4a1b-8632-a23a7d2f2843\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-03362d-5df77d4775-tn9s2" Oct 30 18:59:19.644526 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:19.644500 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8690e67-62d5-4a1b-8632-a23a7d2f2843-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-03362d-5df77d4775-tn9s2\" (UID: \"c8690e67-62d5-4a1b-8632-a23a7d2f2843\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-03362d-5df77d4775-tn9s2" Oct 30 18:59:19.644526 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:19.644519 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5a06a95-d251-46b6-889c-7e7d5e2ca925-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-03362d-655b6cd7f8-ndl7x\" (UID: \"b5a06a95-d251-46b6-889c-7e7d5e2ca925\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-03362d-655b6cd7f8-ndl7x" Oct 30 18:59:19.644840 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:19.644668 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9rwr\" (UniqueName: \"kubernetes.io/projected/b5a06a95-d251-46b6-889c-7e7d5e2ca925-kube-api-access-b9rwr\") pod \"payload-processing-e2e-shared-b-w4-03362d-655b6cd7f8-ndl7x\" (UID: \"b5a06a95-d251-46b6-889c-7e7d5e2ca925\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-03362d-655b6cd7f8-ndl7x" Oct 30 18:59:19.645309 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:19.645286 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5a06a95-d251-46b6-889c-7e7d5e2ca925-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-03362d-655b6cd7f8-ndl7x\" (UID: \"b5a06a95-d251-46b6-889c-7e7d5e2ca925\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-03362d-655b6cd7f8-ndl7x" Oct 30 18:59:19.645465 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:19.645444 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8690e67-62d5-4a1b-8632-a23a7d2f2843-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-03362d-5df77d4775-tn9s2\" (UID: \"c8690e67-62d5-4a1b-8632-a23a7d2f2843\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-03362d-5df77d4775-tn9s2" Oct 30 18:59:19.656834 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:19.656809 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9rwr\" (UniqueName: \"kubernetes.io/projected/b5a06a95-d251-46b6-889c-7e7d5e2ca925-kube-api-access-b9rwr\") pod \"payload-processing-e2e-shared-b-w4-03362d-655b6cd7f8-ndl7x\" (UID: \"b5a06a95-d251-46b6-889c-7e7d5e2ca925\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-03362d-655b6cd7f8-ndl7x" Oct 30 18:59:19.658297 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:19.658271 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpc7q\" (UniqueName: \"kubernetes.io/projected/c8690e67-62d5-4a1b-8632-a23a7d2f2843-kube-api-access-qpc7q\") pod \"payload-pre-processing-e2e-shared-b-w4-03362d-5df77d4775-tn9s2\" (UID: \"c8690e67-62d5-4a1b-8632-a23a7d2f2843\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-03362d-5df77d4775-tn9s2" Oct 30 18:59:19.720257 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:19.720228 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-03362d-5df77d4775-tn9s2" Oct 30 18:59:19.782782 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:19.782747 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-03362d-655b6cd7f8-ndl7x" Oct 30 18:59:20.099690 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:59:20.099663 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8690e67_62d5_4a1b_8632_a23a7d2f2843.slice/crio-da88029127bd3fad84607bb68f45bc54ec3152b32181d6723cc9207da7e5480d WatchSource:0}: Error finding container da88029127bd3fad84607bb68f45bc54ec3152b32181d6723cc9207da7e5480d: Status 404 returned error can't find the container with id da88029127bd3fad84607bb68f45bc54ec3152b32181d6723cc9207da7e5480d Oct 30 18:59:20.100255 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:20.100234 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-03362d-5df77d4775-tn9s2"] Oct 30 18:59:20.100350 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:59:20.100329 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5a06a95_d251_46b6_889c_7e7d5e2ca925.slice/crio-3659bd107e769d2b0cefd9e800cf2bab88f8d6db5dc5ac2a5d68d2465ebd4295 WatchSource:0}: Error finding container 3659bd107e769d2b0cefd9e800cf2bab88f8d6db5dc5ac2a5d68d2465ebd4295: Status 404 returned error can't find the container with id 3659bd107e769d2b0cefd9e800cf2bab88f8d6db5dc5ac2a5d68d2465ebd4295 Oct 30 18:59:20.103502 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:20.103476 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-03362d-655b6cd7f8-ndl7x"] Oct 30 18:59:20.251805 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:20.251768 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:20.337845 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:20.337798 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-03362d-5df77d4775-tn9s2" event={"ID":"c8690e67-62d5-4a1b-8632-a23a7d2f2843","Type":"ContainerStarted","Data":"2de73ed0cef9082202870f240fb4c702b838bc4a07eb49a8f74f61a1eb528621"} Oct 30 18:59:20.338046 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:20.337867 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-03362d-5df77d4775-tn9s2" Oct 30 18:59:20.338046 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:20.337895 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-03362d-5df77d4775-tn9s2" event={"ID":"c8690e67-62d5-4a1b-8632-a23a7d2f2843","Type":"ContainerStarted","Data":"da88029127bd3fad84607bb68f45bc54ec3152b32181d6723cc9207da7e5480d"} Oct 30 18:59:20.349952 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:20.344189 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-03362d-655b6cd7f8-ndl7x" event={"ID":"b5a06a95-d251-46b6-889c-7e7d5e2ca925","Type":"ContainerStarted","Data":"2470337503e5e11c7b374c3739b0d98a86e584f1f3db5fec12207fc2f6d2daca"} Oct 30 18:59:20.349952 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:20.344301 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-03362d-655b6cd7f8-ndl7x" event={"ID":"b5a06a95-d251-46b6-889c-7e7d5e2ca925","Type":"ContainerStarted","Data":"3659bd107e769d2b0cefd9e800cf2bab88f8d6db5dc5ac2a5d68d2465ebd4295"} Oct 30 18:59:20.349952 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:20.344785 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-03362d-655b6cd7f8-ndl7x" Oct 30 18:59:20.367301 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:20.367243 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-03362d-5df77d4775-tn9s2" podStartSLOduration=1.367224484 podStartE2EDuration="1.367224484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:59:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:59:20.363345608 +0000 UTC m=+1527.892376737" watchObservedRunningTime="2026-10-30 18:59:20.367224484 +0000 UTC m=+1527.896255623" Oct 30 18:59:20.391189 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:20.391090 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-03362d-655b6cd7f8-ndl7x" podStartSLOduration=1.391080331 podStartE2EDuration="1.391080331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:59:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:59:20.389350911 +0000 UTC m=+1527.918382050" watchObservedRunningTime="2026-10-30 18:59:20.391080331 +0000 UTC m=+1527.920111470" Oct 30 18:59:20.552057 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:20.552023 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:20.710222 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:20.709674 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:21.546492 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:21.545679 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:21.855745 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:21.855711 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:22.002015 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:22.001981 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:22.623573 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:22.623533 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:23.508537 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:23.508488 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:23.948340 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:59:23.947791 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 18:59:23.982098 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:23.982069 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-vjb5b" Oct 30 18:59:24.032668 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:24.032635 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/160390da-b35c-4030-9fff-d93afbcf9bad-istio-token\" (UniqueName: \"kubernetes.io/projected/160390da-b35c-4030-9fff-d93afbcf9bad-istio-token\") pod \"160390da-b35c-4030-9fff-d93afbcf9bad\" (UID: \"160390da-b35c-4030-9fff-d93afbcf9bad\") " Oct 30 18:59:24.032851 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:24.032694 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/160390da-b35c-4030-9fff-d93afbcf9bad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/160390da-b35c-4030-9fff-d93afbcf9bad-credential-socket\") pod \"160390da-b35c-4030-9fff-d93afbcf9bad\" (UID: \"160390da-b35c-4030-9fff-d93afbcf9bad\") " Oct 30 18:59:24.032851 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:24.032728 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/160390da-b35c-4030-9fff-d93afbcf9bad-kube-api-access-lwvst\" (UniqueName: \"kubernetes.io/projected/160390da-b35c-4030-9fff-d93afbcf9bad-kube-api-access-lwvst\") pod \"160390da-b35c-4030-9fff-d93afbcf9bad\" (UID: \"160390da-b35c-4030-9fff-d93afbcf9bad\") " Oct 30 18:59:24.032851 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:24.032777 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/160390da-b35c-4030-9fff-d93afbcf9bad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/160390da-b35c-4030-9fff-d93afbcf9bad-istiod-ca-cert\") pod \"160390da-b35c-4030-9fff-d93afbcf9bad\" (UID: \"160390da-b35c-4030-9fff-d93afbcf9bad\") " Oct 30 18:59:24.032851 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:24.032831 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/160390da-b35c-4030-9fff-d93afbcf9bad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/160390da-b35c-4030-9fff-d93afbcf9bad-workload-socket\") pod \"160390da-b35c-4030-9fff-d93afbcf9bad\" (UID: \"160390da-b35c-4030-9fff-d93afbcf9bad\") " Oct 30 18:59:24.033085 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:24.032894 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/160390da-b35c-4030-9fff-d93afbcf9bad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/160390da-b35c-4030-9fff-d93afbcf9bad-workload-certs\") pod \"160390da-b35c-4030-9fff-d93afbcf9bad\" (UID: \"160390da-b35c-4030-9fff-d93afbcf9bad\") " Oct 30 18:59:24.033085 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:24.032952 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/160390da-b35c-4030-9fff-d93afbcf9bad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/160390da-b35c-4030-9fff-d93afbcf9bad-istio-envoy\") pod \"160390da-b35c-4030-9fff-d93afbcf9bad\" (UID: \"160390da-b35c-4030-9fff-d93afbcf9bad\") " Oct 30 18:59:24.033085 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:24.032997 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/160390da-b35c-4030-9fff-d93afbcf9bad-credential-socket" pod "160390da-b35c-4030-9fff-d93afbcf9bad" (UID: "160390da-b35c-4030-9fff-d93afbcf9bad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:59:24.033085 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:24.033007 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/160390da-b35c-4030-9fff-d93afbcf9bad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/160390da-b35c-4030-9fff-d93afbcf9bad-istio-podinfo\") pod \"160390da-b35c-4030-9fff-d93afbcf9bad\" (UID: \"160390da-b35c-4030-9fff-d93afbcf9bad\") " Oct 30 18:59:24.033085 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:24.033076 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/160390da-b35c-4030-9fff-d93afbcf9bad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/160390da-b35c-4030-9fff-d93afbcf9bad-istio-data\") pod \"160390da-b35c-4030-9fff-d93afbcf9bad\" (UID: \"160390da-b35c-4030-9fff-d93afbcf9bad\") " Oct 30 18:59:24.033408 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:24.033394 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/160390da-b35c-4030-9fff-d93afbcf9bad-workload-socket" pod "160390da-b35c-4030-9fff-d93afbcf9bad" (UID: "160390da-b35c-4030-9fff-d93afbcf9bad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:59:24.033623 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:24.033600 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/160390da-b35c-4030-9fff-d93afbcf9bad-istio-data" pod "160390da-b35c-4030-9fff-d93afbcf9bad" (UID: "160390da-b35c-4030-9fff-d93afbcf9bad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:59:24.034373 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:24.033794 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/160390da-b35c-4030-9fff-d93afbcf9bad-workload-certs" pod "160390da-b35c-4030-9fff-d93afbcf9bad" (UID: "160390da-b35c-4030-9fff-d93afbcf9bad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:59:24.034373 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:24.034050 6960 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/160390da-b35c-4030-9fff-d93afbcf9bad-istio-data\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:59:24.034373 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:24.034175 6960 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/160390da-b35c-4030-9fff-d93afbcf9bad-credential-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:59:24.034373 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:24.034195 6960 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/160390da-b35c-4030-9fff-d93afbcf9bad-workload-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:59:24.034373 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:24.034211 6960 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/160390da-b35c-4030-9fff-d93afbcf9bad-workload-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:59:24.034708 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:24.034400 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/160390da-b35c-4030-9fff-d93afbcf9bad-istiod-ca-cert" pod "160390da-b35c-4030-9fff-d93afbcf9bad" (UID: "160390da-b35c-4030-9fff-d93afbcf9bad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:59:24.038968 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:24.038897 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/160390da-b35c-4030-9fff-d93afbcf9bad-istio-podinfo" pod "160390da-b35c-4030-9fff-d93afbcf9bad" (UID: "160390da-b35c-4030-9fff-d93afbcf9bad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:59:24.039108 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:24.038994 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/160390da-b35c-4030-9fff-d93afbcf9bad-kube-api-access-lwvst" pod "160390da-b35c-4030-9fff-d93afbcf9bad" (UID: "160390da-b35c-4030-9fff-d93afbcf9bad"). InnerVolumeSpecName "kube-api-access-lwvst". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:59:24.039471 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:24.039452 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/160390da-b35c-4030-9fff-d93afbcf9bad-istio-token" pod "160390da-b35c-4030-9fff-d93afbcf9bad" (UID: "160390da-b35c-4030-9fff-d93afbcf9bad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:59:24.039997 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:24.039981 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/160390da-b35c-4030-9fff-d93afbcf9bad-istio-envoy" pod "160390da-b35c-4030-9fff-d93afbcf9bad" (UID: "160390da-b35c-4030-9fff-d93afbcf9bad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:59:24.065568 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:24.065548 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-n9mh9" Oct 30 18:59:24.135451 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:24.135421 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6eaab09d-e908-4663-9793-30cff7591dfb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eaab09d-e908-4663-9793-30cff7591dfb-credential-socket\") pod \"6eaab09d-e908-4663-9793-30cff7591dfb\" (UID: \"6eaab09d-e908-4663-9793-30cff7591dfb\") " Oct 30 18:59:24.135451 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:24.135454 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6eaab09d-e908-4663-9793-30cff7591dfb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6eaab09d-e908-4663-9793-30cff7591dfb-workload-certs\") pod \"6eaab09d-e908-4663-9793-30cff7591dfb\" (UID: \"6eaab09d-e908-4663-9793-30cff7591dfb\") " Oct 30 18:59:24.135700 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:24.135489 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6eaab09d-e908-4663-9793-30cff7591dfb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6eaab09d-e908-4663-9793-30cff7591dfb-istio-envoy\") pod \"6eaab09d-e908-4663-9793-30cff7591dfb\" (UID: \"6eaab09d-e908-4663-9793-30cff7591dfb\") " Oct 30 18:59:24.135700 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:24.135594 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6eaab09d-e908-4663-9793-30cff7591dfb-kube-api-access-4hnlz\" (UniqueName: \"kubernetes.io/projected/6eaab09d-e908-4663-9793-30cff7591dfb-kube-api-access-4hnlz\") pod \"6eaab09d-e908-4663-9793-30cff7591dfb\" (UID: \"6eaab09d-e908-4663-9793-30cff7591dfb\") " Oct 30 18:59:24.135700 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:24.135687 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6eaab09d-e908-4663-9793-30cff7591dfb-istio-token\" (UniqueName: \"kubernetes.io/projected/6eaab09d-e908-4663-9793-30cff7591dfb-istio-token\") pod \"6eaab09d-e908-4663-9793-30cff7591dfb\" (UID: \"6eaab09d-e908-4663-9793-30cff7591dfb\") " Oct 30 18:59:24.135864 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:24.135745 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6eaab09d-e908-4663-9793-30cff7591dfb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6eaab09d-e908-4663-9793-30cff7591dfb-istio-data\") pod \"6eaab09d-e908-4663-9793-30cff7591dfb\" (UID: \"6eaab09d-e908-4663-9793-30cff7591dfb\") " Oct 30 18:59:24.135864 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:24.135818 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6eaab09d-e908-4663-9793-30cff7591dfb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6eaab09d-e908-4663-9793-30cff7591dfb-istio-podinfo\") pod \"6eaab09d-e908-4663-9793-30cff7591dfb\" (UID: \"6eaab09d-e908-4663-9793-30cff7591dfb\") " Oct 30 18:59:24.135864 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:24.135847 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6eaab09d-e908-4663-9793-30cff7591dfb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eaab09d-e908-4663-9793-30cff7591dfb-workload-socket\") pod \"6eaab09d-e908-4663-9793-30cff7591dfb\" (UID: \"6eaab09d-e908-4663-9793-30cff7591dfb\") " Oct 30 18:59:24.136025 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:24.135967 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6eaab09d-e908-4663-9793-30cff7591dfb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6eaab09d-e908-4663-9793-30cff7591dfb-istiod-ca-cert\") pod \"6eaab09d-e908-4663-9793-30cff7591dfb\" (UID: \"6eaab09d-e908-4663-9793-30cff7591dfb\") " Oct 30 18:59:24.136025 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:24.135739 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6eaab09d-e908-4663-9793-30cff7591dfb-credential-socket" pod "6eaab09d-e908-4663-9793-30cff7591dfb" (UID: "6eaab09d-e908-4663-9793-30cff7591dfb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:59:24.136119 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:24.135784 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6eaab09d-e908-4663-9793-30cff7591dfb-workload-certs" pod "6eaab09d-e908-4663-9793-30cff7591dfb" (UID: "6eaab09d-e908-4663-9793-30cff7591dfb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:59:24.136119 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:24.135996 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6eaab09d-e908-4663-9793-30cff7591dfb-istio-data" pod "6eaab09d-e908-4663-9793-30cff7591dfb" (UID: "6eaab09d-e908-4663-9793-30cff7591dfb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:59:24.136307 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:24.136259 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6eaab09d-e908-4663-9793-30cff7591dfb-workload-socket" pod "6eaab09d-e908-4663-9793-30cff7591dfb" (UID: "6eaab09d-e908-4663-9793-30cff7591dfb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:59:24.136484 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:24.136457 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6eaab09d-e908-4663-9793-30cff7591dfb-istiod-ca-cert" pod "6eaab09d-e908-4663-9793-30cff7591dfb" (UID: "6eaab09d-e908-4663-9793-30cff7591dfb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:59:24.136914 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:24.136838 6960 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/160390da-b35c-4030-9fff-d93afbcf9bad-istio-token\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:59:24.136914 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:24.136868 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwvst\" (UniqueName: \"kubernetes.io/projected/160390da-b35c-4030-9fff-d93afbcf9bad-kube-api-access-lwvst\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:59:24.136914 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:24.136886 6960 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/160390da-b35c-4030-9fff-d93afbcf9bad-istiod-ca-cert\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:59:24.136914 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:24.136901 6960 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6eaab09d-e908-4663-9793-30cff7591dfb-istio-data\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:59:24.136914 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:24.136918 6960 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eaab09d-e908-4663-9793-30cff7591dfb-workload-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:59:24.137327 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:24.136931 6960 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6eaab09d-e908-4663-9793-30cff7591dfb-istiod-ca-cert\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:59:24.137327 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:24.136945 6960 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/160390da-b35c-4030-9fff-d93afbcf9bad-istio-envoy\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:59:24.137327 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:24.136958 6960 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eaab09d-e908-4663-9793-30cff7591dfb-credential-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:59:24.137327 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:24.136971 6960 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6eaab09d-e908-4663-9793-30cff7591dfb-workload-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:59:24.137327 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:24.136984 6960 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/160390da-b35c-4030-9fff-d93afbcf9bad-istio-podinfo\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:59:24.138030 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:24.138004 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6eaab09d-e908-4663-9793-30cff7591dfb-istio-token" pod "6eaab09d-e908-4663-9793-30cff7591dfb" (UID: "6eaab09d-e908-4663-9793-30cff7591dfb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:59:24.138137 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:24.138033 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6eaab09d-e908-4663-9793-30cff7591dfb-istio-envoy" pod "6eaab09d-e908-4663-9793-30cff7591dfb" (UID: "6eaab09d-e908-4663-9793-30cff7591dfb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:59:24.138551 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:24.138533 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6eaab09d-e908-4663-9793-30cff7591dfb-kube-api-access-4hnlz" pod "6eaab09d-e908-4663-9793-30cff7591dfb" (UID: "6eaab09d-e908-4663-9793-30cff7591dfb"). InnerVolumeSpecName "kube-api-access-4hnlz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:59:24.138855 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:24.138832 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6eaab09d-e908-4663-9793-30cff7591dfb-istio-podinfo" pod "6eaab09d-e908-4663-9793-30cff7591dfb" (UID: "6eaab09d-e908-4663-9793-30cff7591dfb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:59:24.238321 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:24.238293 6960 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6eaab09d-e908-4663-9793-30cff7591dfb-istio-envoy\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:59:24.238321 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:24.238319 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hnlz\" (UniqueName: \"kubernetes.io/projected/6eaab09d-e908-4663-9793-30cff7591dfb-kube-api-access-4hnlz\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:59:24.238321 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:24.238329 6960 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6eaab09d-e908-4663-9793-30cff7591dfb-istio-token\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:59:24.238549 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:24.238338 6960 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6eaab09d-e908-4663-9793-30cff7591dfb-istio-podinfo\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 18:59:24.408195 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:24.408014 6960 generic.go:356] "Generic (PLEG): container finished" podID="6eaab09d-e908-4663-9793-30cff7591dfb" containerID="efb91e4d840cdee5f09dd66ad3266a4675fbf390be43ebefd7e21c85c9fca6bd" exitCode=0 Oct 30 18:59:24.412246 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:24.408468 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-n9mh9" Oct 30 18:59:24.412812 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:24.412471 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-n9mh9" event={"ID":"6eaab09d-e908-4663-9793-30cff7591dfb","Type":"ContainerDied","Data":"efb91e4d840cdee5f09dd66ad3266a4675fbf390be43ebefd7e21c85c9fca6bd"} Oct 30 18:59:24.412812 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:24.412585 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-n9mh9" event={"ID":"6eaab09d-e908-4663-9793-30cff7591dfb","Type":"ContainerDied","Data":"0344cdebe400b85c8489135c1339a7b8c80246d30b4943d3fb2f92ee365a34f7"} Oct 30 18:59:24.412812 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:24.412618 6960 scope.go:122] "RemoveContainer" containerID="efb91e4d840cdee5f09dd66ad3266a4675fbf390be43ebefd7e21c85c9fca6bd" Oct 30 18:59:24.420081 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:24.419903 6960 generic.go:356] "Generic (PLEG): container finished" podID="160390da-b35c-4030-9fff-d93afbcf9bad" containerID="af04992b96a297bcf36ebea4a331cfe69c8e368f7ccc7012f4e8851315113726" exitCode=0 Oct 30 18:59:24.420081 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:24.419991 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-vjb5b" event={"ID":"160390da-b35c-4030-9fff-d93afbcf9bad","Type":"ContainerDied","Data":"af04992b96a297bcf36ebea4a331cfe69c8e368f7ccc7012f4e8851315113726"} Oct 30 18:59:24.420081 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:24.420038 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-vjb5b" event={"ID":"160390da-b35c-4030-9fff-d93afbcf9bad","Type":"ContainerDied","Data":"64aecded2197f325b034fb18115e96ac6a5a0db79c41aee7f79e303a12576896"} Oct 30 18:59:24.420377 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:24.420090 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-vjb5b" Oct 30 18:59:24.443863 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:24.442189 6960 scope.go:122] "RemoveContainer" containerID="efb91e4d840cdee5f09dd66ad3266a4675fbf390be43ebefd7e21c85c9fca6bd" Oct 30 18:59:24.444965 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:59:24.444595 6960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efb91e4d840cdee5f09dd66ad3266a4675fbf390be43ebefd7e21c85c9fca6bd\": container with ID starting with efb91e4d840cdee5f09dd66ad3266a4675fbf390be43ebefd7e21c85c9fca6bd not found: ID does not exist" containerID="efb91e4d840cdee5f09dd66ad3266a4675fbf390be43ebefd7e21c85c9fca6bd" Oct 30 18:59:24.444965 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:24.444632 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efb91e4d840cdee5f09dd66ad3266a4675fbf390be43ebefd7e21c85c9fca6bd"} err="failed to get container status \"efb91e4d840cdee5f09dd66ad3266a4675fbf390be43ebefd7e21c85c9fca6bd\": rpc error: code = NotFound desc = could not find container \"efb91e4d840cdee5f09dd66ad3266a4675fbf390be43ebefd7e21c85c9fca6bd\": container with ID starting with efb91e4d840cdee5f09dd66ad3266a4675fbf390be43ebefd7e21c85c9fca6bd not found: ID does not exist" Oct 30 18:59:24.444965 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:24.444659 6960 scope.go:122] "RemoveContainer" containerID="af04992b96a297bcf36ebea4a331cfe69c8e368f7ccc7012f4e8851315113726" Oct 30 18:59:24.447887 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:24.447611 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:24.476897 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:24.474434 6960 scope.go:122] "RemoveContainer" containerID="af04992b96a297bcf36ebea4a331cfe69c8e368f7ccc7012f4e8851315113726" Oct 30 18:59:24.476897 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:59:24.474868 6960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af04992b96a297bcf36ebea4a331cfe69c8e368f7ccc7012f4e8851315113726\": container with ID starting with af04992b96a297bcf36ebea4a331cfe69c8e368f7ccc7012f4e8851315113726 not found: ID does not exist" containerID="af04992b96a297bcf36ebea4a331cfe69c8e368f7ccc7012f4e8851315113726" Oct 30 18:59:24.476897 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:24.474903 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af04992b96a297bcf36ebea4a331cfe69c8e368f7ccc7012f4e8851315113726"} err="failed to get container status \"af04992b96a297bcf36ebea4a331cfe69c8e368f7ccc7012f4e8851315113726\": rpc error: code = NotFound desc = could not find container \"af04992b96a297bcf36ebea4a331cfe69c8e368f7ccc7012f4e8851315113726\": container with ID starting with af04992b96a297bcf36ebea4a331cfe69c8e368f7ccc7012f4e8851315113726 not found: ID does not exist" Oct 30 18:59:24.477332 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:24.477273 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-vjb5b"] Oct 30 18:59:24.480920 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:24.480897 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-vjb5b"] Oct 30 18:59:24.492238 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:24.492218 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-n9mh9"] Oct 30 18:59:24.496721 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:24.496694 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-n9mh9"] Oct 30 18:59:24.595435 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:24.595398 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="160390da-b35c-4030-9fff-d93afbcf9bad" path="/var/lib/kubelet/pods/160390da-b35c-4030-9fff-d93afbcf9bad/volumes" Oct 30 18:59:24.596171 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:24.596137 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6eaab09d-e908-4663-9793-30cff7591dfb" path="/var/lib/kubelet/pods/6eaab09d-e908-4663-9793-30cff7591dfb/volumes" Oct 30 18:59:25.485428 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:25.485390 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:26.327568 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:26.327531 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:26.462012 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:26.461975 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:26.722062 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:26.721975 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-gtlvx"] Oct 30 18:59:26.722779 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:26.722755 6960 cpu_manager.go:414] "containerMap: removing container" podUID="6eaab09d-e908-4663-9793-30cff7591dfb" containerName="istio-proxy" Oct 30 18:59:26.722914 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:26.722781 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6eaab09d-e908-4663-9793-30cff7591dfb" containerName="istio-proxy" Oct 30 18:59:26.722914 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:26.722836 6960 cpu_manager.go:414] "containerMap: removing container" podUID="160390da-b35c-4030-9fff-d93afbcf9bad" containerName="istio-proxy" Oct 30 18:59:26.722914 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:26.722845 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="160390da-b35c-4030-9fff-d93afbcf9bad" containerName="istio-proxy" Oct 30 18:59:26.723072 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:26.723049 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="160390da-b35c-4030-9fff-d93afbcf9bad" containerName="istio-proxy" Oct 30 18:59:26.723126 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:26.723078 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="6eaab09d-e908-4663-9793-30cff7591dfb" containerName="istio-proxy" Oct 30 18:59:26.728752 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:26.728729 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-gtlvx" Oct 30 18:59:26.741464 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:26.741441 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-gtlvx"] Oct 30 18:59:26.792908 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:26.792882 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c4cb031-1d4b-4026-8846-bcf16e2b5972-istio-podinfo\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-gtlvx\" (UID: \"2c4cb031-1d4b-4026-8846-bcf16e2b5972\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-gtlvx" Oct 30 18:59:26.793036 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:26.792941 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c4cb031-1d4b-4026-8846-bcf16e2b5972-istio-envoy\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-gtlvx\" (UID: \"2c4cb031-1d4b-4026-8846-bcf16e2b5972\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-gtlvx" Oct 30 18:59:26.793088 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:26.793031 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c4cb031-1d4b-4026-8846-bcf16e2b5972-istio-token\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-gtlvx\" (UID: \"2c4cb031-1d4b-4026-8846-bcf16e2b5972\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-gtlvx" Oct 30 18:59:26.793088 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:26.793059 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c4cb031-1d4b-4026-8846-bcf16e2b5972-workload-socket\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-gtlvx\" (UID: \"2c4cb031-1d4b-4026-8846-bcf16e2b5972\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-gtlvx" Oct 30 18:59:26.793088 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:26.793076 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c4cb031-1d4b-4026-8846-bcf16e2b5972-credential-socket\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-gtlvx\" (UID: \"2c4cb031-1d4b-4026-8846-bcf16e2b5972\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-gtlvx" Oct 30 18:59:26.793226 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:26.793095 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jnx6\" (UniqueName: \"kubernetes.io/projected/2c4cb031-1d4b-4026-8846-bcf16e2b5972-kube-api-access-6jnx6\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-gtlvx\" (UID: \"2c4cb031-1d4b-4026-8846-bcf16e2b5972\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-gtlvx" Oct 30 18:59:26.793282 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:26.793266 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c4cb031-1d4b-4026-8846-bcf16e2b5972-istio-data\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-gtlvx\" (UID: \"2c4cb031-1d4b-4026-8846-bcf16e2b5972\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-gtlvx" Oct 30 18:59:26.793321 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:26.793312 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c4cb031-1d4b-4026-8846-bcf16e2b5972-istiod-ca-cert\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-gtlvx\" (UID: \"2c4cb031-1d4b-4026-8846-bcf16e2b5972\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-gtlvx" Oct 30 18:59:26.793355 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:26.793334 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c4cb031-1d4b-4026-8846-bcf16e2b5972-workload-certs\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-gtlvx\" (UID: \"2c4cb031-1d4b-4026-8846-bcf16e2b5972\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-gtlvx" Oct 30 18:59:26.894429 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:26.894402 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c4cb031-1d4b-4026-8846-bcf16e2b5972-istiod-ca-cert\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-gtlvx\" (UID: \"2c4cb031-1d4b-4026-8846-bcf16e2b5972\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-gtlvx" Oct 30 18:59:26.894429 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:26.894434 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c4cb031-1d4b-4026-8846-bcf16e2b5972-workload-certs\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-gtlvx\" (UID: \"2c4cb031-1d4b-4026-8846-bcf16e2b5972\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-gtlvx" Oct 30 18:59:26.894613 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:26.894531 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c4cb031-1d4b-4026-8846-bcf16e2b5972-istio-podinfo\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-gtlvx\" (UID: \"2c4cb031-1d4b-4026-8846-bcf16e2b5972\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-gtlvx" Oct 30 18:59:26.894613 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:26.894579 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c4cb031-1d4b-4026-8846-bcf16e2b5972-istio-envoy\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-gtlvx\" (UID: \"2c4cb031-1d4b-4026-8846-bcf16e2b5972\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-gtlvx" Oct 30 18:59:26.894613 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:26.894601 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c4cb031-1d4b-4026-8846-bcf16e2b5972-istio-token\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-gtlvx\" (UID: \"2c4cb031-1d4b-4026-8846-bcf16e2b5972\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-gtlvx" Oct 30 18:59:26.894752 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:26.894629 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c4cb031-1d4b-4026-8846-bcf16e2b5972-workload-socket\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-gtlvx\" (UID: \"2c4cb031-1d4b-4026-8846-bcf16e2b5972\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-gtlvx" Oct 30 18:59:26.894752 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:26.894653 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c4cb031-1d4b-4026-8846-bcf16e2b5972-credential-socket\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-gtlvx\" (UID: \"2c4cb031-1d4b-4026-8846-bcf16e2b5972\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-gtlvx" Oct 30 18:59:26.894752 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:26.894680 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jnx6\" (UniqueName: \"kubernetes.io/projected/2c4cb031-1d4b-4026-8846-bcf16e2b5972-kube-api-access-6jnx6\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-gtlvx\" (UID: \"2c4cb031-1d4b-4026-8846-bcf16e2b5972\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-gtlvx" Oct 30 18:59:26.894752 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:26.894748 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c4cb031-1d4b-4026-8846-bcf16e2b5972-istio-data\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-gtlvx\" (UID: \"2c4cb031-1d4b-4026-8846-bcf16e2b5972\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-gtlvx" Oct 30 18:59:26.894953 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:26.894935 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c4cb031-1d4b-4026-8846-bcf16e2b5972-workload-certs\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-gtlvx\" (UID: \"2c4cb031-1d4b-4026-8846-bcf16e2b5972\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-gtlvx" Oct 30 18:59:26.895105 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:26.895078 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c4cb031-1d4b-4026-8846-bcf16e2b5972-istio-data\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-gtlvx\" (UID: \"2c4cb031-1d4b-4026-8846-bcf16e2b5972\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-gtlvx" Oct 30 18:59:26.895226 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:26.895137 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c4cb031-1d4b-4026-8846-bcf16e2b5972-credential-socket\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-gtlvx\" (UID: \"2c4cb031-1d4b-4026-8846-bcf16e2b5972\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-gtlvx" Oct 30 18:59:26.895226 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:26.895202 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c4cb031-1d4b-4026-8846-bcf16e2b5972-istiod-ca-cert\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-gtlvx\" (UID: \"2c4cb031-1d4b-4026-8846-bcf16e2b5972\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-gtlvx" Oct 30 18:59:26.895566 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:26.895279 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c4cb031-1d4b-4026-8846-bcf16e2b5972-workload-socket\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-gtlvx\" (UID: \"2c4cb031-1d4b-4026-8846-bcf16e2b5972\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-gtlvx" Oct 30 18:59:26.897039 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:26.897016 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c4cb031-1d4b-4026-8846-bcf16e2b5972-istio-envoy\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-gtlvx\" (UID: \"2c4cb031-1d4b-4026-8846-bcf16e2b5972\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-gtlvx" Oct 30 18:59:26.897232 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:26.897217 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c4cb031-1d4b-4026-8846-bcf16e2b5972-istio-podinfo\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-gtlvx\" (UID: \"2c4cb031-1d4b-4026-8846-bcf16e2b5972\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-gtlvx" Oct 30 18:59:26.906430 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:26.906411 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c4cb031-1d4b-4026-8846-bcf16e2b5972-istio-token\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-gtlvx\" (UID: \"2c4cb031-1d4b-4026-8846-bcf16e2b5972\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-gtlvx" Oct 30 18:59:26.906740 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:26.906723 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jnx6\" (UniqueName: \"kubernetes.io/projected/2c4cb031-1d4b-4026-8846-bcf16e2b5972-kube-api-access-6jnx6\") pod \"e2e-worker-w1-aa6736-openshift-default-6f74f55675-gtlvx\" (UID: \"2c4cb031-1d4b-4026-8846-bcf16e2b5972\") " pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-gtlvx" Oct 30 18:59:27.047273 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:27.047177 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-gtlvx" Oct 30 18:59:27.227889 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:27.227847 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:27.334644 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:27.334620 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-gtlvx"] Oct 30 18:59:27.338766 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:59:27.338282 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c4cb031_1d4b_4026_8846_bcf16e2b5972.slice/crio-f538e4b29462cb222a00369d4b7e3f38fd198dd45e8316c9f6cc4607983461a8 WatchSource:0}: Error finding container f538e4b29462cb222a00369d4b7e3f38fd198dd45e8316c9f6cc4607983461a8: Status 404 returned error can't find the container with id f538e4b29462cb222a00369d4b7e3f38fd198dd45e8316c9f6cc4607983461a8 Oct 30 18:59:27.341202 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:27.341168 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:59:27.341327 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:27.341231 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:59:27.341327 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:27.341260 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 18:59:27.445832 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:27.445793 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-gtlvx" event={"ID":"2c4cb031-1d4b-4026-8846-bcf16e2b5972","Type":"ContainerStarted","Data":"90a64aed40ad340601508841f812164bbbc9a9e60ce1d5156f5bd51d3dfbe9c1"} Oct 30 18:59:27.446016 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:27.445853 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-gtlvx" event={"ID":"2c4cb031-1d4b-4026-8846-bcf16e2b5972","Type":"ContainerStarted","Data":"f538e4b29462cb222a00369d4b7e3f38fd198dd45e8316c9f6cc4607983461a8"} Oct 30 18:59:27.472125 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:27.472069 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-gtlvx" podStartSLOduration=1.472050684 podStartE2EDuration="1.472050684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:59:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:59:27.470066621 +0000 UTC m=+1534.999097774" watchObservedRunningTime="2026-10-30 18:59:27.472050684 +0000 UTC m=+1535.001081826" Oct 30 18:59:28.047469 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:28.047433 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-gtlvx" Oct 30 18:59:28.049171 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:28.049128 6960 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-gtlvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 30 18:59:28.049299 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:28.049202 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-gtlvx" podUID="2c4cb031-1d4b-4026-8846-bcf16e2b5972" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 30 18:59:28.277174 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:28.272844 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:29.050196 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:29.049528 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-03362d-6fdbdf99fb-72dp5"] Oct 30 18:59:29.051243 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:29.051209 6960 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-gtlvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 30 18:59:29.051393 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:29.051263 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-gtlvx" podUID="2c4cb031-1d4b-4026-8846-bcf16e2b5972" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 30 18:59:29.085525 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:29.085437 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-03362d-6fdbdf99fb-72dp5" Oct 30 18:59:29.091600 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:29.091571 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-03362d-6fdbdf99fb-72dp5"] Oct 30 18:59:29.095450 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:29.095416 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-03362d-serving-cert\"" Oct 30 18:59:29.120065 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:29.120031 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:29.127804 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:29.127779 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/efd98aa3-a350-4bf9-82ec-34f7b7917adc-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-03362d-6fdbdf99fb-72dp5\" (UID: \"efd98aa3-a350-4bf9-82ec-34f7b7917adc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-03362d-6fdbdf99fb-72dp5" Oct 30 18:59:29.127932 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:29.127814 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/efd98aa3-a350-4bf9-82ec-34f7b7917adc-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-03362d-6fdbdf99fb-72dp5\" (UID: \"efd98aa3-a350-4bf9-82ec-34f7b7917adc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-03362d-6fdbdf99fb-72dp5" Oct 30 18:59:29.127979 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:29.127953 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/efd98aa3-a350-4bf9-82ec-34f7b7917adc-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-03362d-6fdbdf99fb-72dp5\" (UID: \"efd98aa3-a350-4bf9-82ec-34f7b7917adc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-03362d-6fdbdf99fb-72dp5" Oct 30 18:59:29.128178 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:29.128134 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbv87\" (UniqueName: \"kubernetes.io/projected/efd98aa3-a350-4bf9-82ec-34f7b7917adc-kube-api-access-cbv87\") pod \"maas-api-e2e-shared-b-w4-03362d-6fdbdf99fb-72dp5\" (UID: \"efd98aa3-a350-4bf9-82ec-34f7b7917adc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-03362d-6fdbdf99fb-72dp5" Oct 30 18:59:29.229245 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:29.229208 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbv87\" (UniqueName: \"kubernetes.io/projected/efd98aa3-a350-4bf9-82ec-34f7b7917adc-kube-api-access-cbv87\") pod \"maas-api-e2e-shared-b-w4-03362d-6fdbdf99fb-72dp5\" (UID: \"efd98aa3-a350-4bf9-82ec-34f7b7917adc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-03362d-6fdbdf99fb-72dp5" Oct 30 18:59:29.229433 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:29.229292 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/efd98aa3-a350-4bf9-82ec-34f7b7917adc-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-03362d-6fdbdf99fb-72dp5\" (UID: \"efd98aa3-a350-4bf9-82ec-34f7b7917adc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-03362d-6fdbdf99fb-72dp5" Oct 30 18:59:29.229433 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:29.229319 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/efd98aa3-a350-4bf9-82ec-34f7b7917adc-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-03362d-6fdbdf99fb-72dp5\" (UID: \"efd98aa3-a350-4bf9-82ec-34f7b7917adc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-03362d-6fdbdf99fb-72dp5" Oct 30 18:59:29.229541 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:29.229430 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/efd98aa3-a350-4bf9-82ec-34f7b7917adc-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-03362d-6fdbdf99fb-72dp5\" (UID: \"efd98aa3-a350-4bf9-82ec-34f7b7917adc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-03362d-6fdbdf99fb-72dp5" Oct 30 18:59:29.229541 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:59:29.229528 6960 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-03362d-serving-cert: secret "maas-api-e2e-shared-b-w4-03362d-serving-cert" not found Oct 30 18:59:29.229644 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:59:29.229609 6960 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/efd98aa3-a350-4bf9-82ec-34f7b7917adc-maas-api-tls podName:efd98aa3-a350-4bf9-82ec-34f7b7917adc nodeName:}" failed. No retries permitted until 2026-10-30 18:59:29.729586787 +0000 UTC m=+1537.258617917 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/efd98aa3-a350-4bf9-82ec-34f7b7917adc-maas-api-tls") pod "maas-api-e2e-shared-b-w4-03362d-6fdbdf99fb-72dp5" (UID: "efd98aa3-a350-4bf9-82ec-34f7b7917adc") : secret "maas-api-e2e-shared-b-w4-03362d-serving-cert" not found Oct 30 18:59:29.229983 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:29.229962 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/efd98aa3-a350-4bf9-82ec-34f7b7917adc-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-03362d-6fdbdf99fb-72dp5\" (UID: \"efd98aa3-a350-4bf9-82ec-34f7b7917adc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-03362d-6fdbdf99fb-72dp5" Oct 30 18:59:29.232163 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:29.232115 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/efd98aa3-a350-4bf9-82ec-34f7b7917adc-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-03362d-6fdbdf99fb-72dp5\" (UID: \"efd98aa3-a350-4bf9-82ec-34f7b7917adc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-03362d-6fdbdf99fb-72dp5" Oct 30 18:59:29.242222 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:29.242194 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:29.248026 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:29.248007 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbv87\" (UniqueName: \"kubernetes.io/projected/efd98aa3-a350-4bf9-82ec-34f7b7917adc-kube-api-access-cbv87\") pod \"maas-api-e2e-shared-b-w4-03362d-6fdbdf99fb-72dp5\" (UID: \"efd98aa3-a350-4bf9-82ec-34f7b7917adc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-03362d-6fdbdf99fb-72dp5" Oct 30 18:59:29.740533 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:29.740496 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/efd98aa3-a350-4bf9-82ec-34f7b7917adc-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-03362d-6fdbdf99fb-72dp5\" (UID: \"efd98aa3-a350-4bf9-82ec-34f7b7917adc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-03362d-6fdbdf99fb-72dp5" Oct 30 18:59:29.740733 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:59:29.740653 6960 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-03362d-serving-cert: secret "maas-api-e2e-shared-b-w4-03362d-serving-cert" not found Oct 30 18:59:29.740733 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:59:29.740729 6960 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/efd98aa3-a350-4bf9-82ec-34f7b7917adc-maas-api-tls podName:efd98aa3-a350-4bf9-82ec-34f7b7917adc nodeName:}" failed. No retries permitted until 2026-10-30 18:59:30.740711595 +0000 UTC m=+1538.269742724 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/efd98aa3-a350-4bf9-82ec-34f7b7917adc-maas-api-tls") pod "maas-api-e2e-shared-b-w4-03362d-6fdbdf99fb-72dp5" (UID: "efd98aa3-a350-4bf9-82ec-34f7b7917adc") : secret "maas-api-e2e-shared-b-w4-03362d-serving-cert" not found Oct 30 18:59:29.894669 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:29.894627 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:30.047830 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:30.047741 6960 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-gtlvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 30 18:59:30.047830 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:30.047796 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-gtlvx" podUID="2c4cb031-1d4b-4026-8846-bcf16e2b5972" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 30 18:59:30.104167 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:30.098312 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:30.233189 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:30.233154 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:30.338873 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:30.338812 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:30.682418 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:30.682327 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:30.779172 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:30.779081 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/efd98aa3-a350-4bf9-82ec-34f7b7917adc-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-03362d-6fdbdf99fb-72dp5\" (UID: \"efd98aa3-a350-4bf9-82ec-34f7b7917adc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-03362d-6fdbdf99fb-72dp5" Oct 30 18:59:30.781899 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:30.781874 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/efd98aa3-a350-4bf9-82ec-34f7b7917adc-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-03362d-6fdbdf99fb-72dp5\" (UID: \"efd98aa3-a350-4bf9-82ec-34f7b7917adc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-03362d-6fdbdf99fb-72dp5" Oct 30 18:59:30.929169 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:30.925408 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-03362d-6fdbdf99fb-72dp5" Oct 30 18:59:31.049166 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:31.048351 6960 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-gtlvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 30 18:59:31.049166 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:31.048547 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-gtlvx" podUID="2c4cb031-1d4b-4026-8846-bcf16e2b5972" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 30 18:59:31.195917 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:31.195002 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:31.463202 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:59:31.455421 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefd98aa3_a350_4bf9_82ec_34f7b7917adc.slice/crio-9465bee0a705addb914204a4b0e6ebabf16d076d41940d76b08d2cb06e2d54bf WatchSource:0}: Error finding container 9465bee0a705addb914204a4b0e6ebabf16d076d41940d76b08d2cb06e2d54bf: Status 404 returned error can't find the container with id 9465bee0a705addb914204a4b0e6ebabf16d076d41940d76b08d2cb06e2d54bf Oct 30 18:59:31.463202 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:31.458259 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-03362d-655b6cd7f8-ndl7x" Oct 30 18:59:31.474249 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:31.464913 6960 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 18:59:31.475477 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:31.474508 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-03362d-6fdbdf99fb-72dp5"] Oct 30 18:59:31.484589 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:31.481208 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-03362d-5df77d4775-tn9s2" Oct 30 18:59:31.549704 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:31.549665 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-03362d-6fdbdf99fb-72dp5" event={"ID":"efd98aa3-a350-4bf9-82ec-34f7b7917adc","Type":"ContainerStarted","Data":"9465bee0a705addb914204a4b0e6ebabf16d076d41940d76b08d2cb06e2d54bf"} Oct 30 18:59:31.707503 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:31.707402 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:31.982313 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:31.982232 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:32.047493 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:32.047460 6960 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-gtlvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 30 18:59:32.047704 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:32.047515 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-gtlvx" podUID="2c4cb031-1d4b-4026-8846-bcf16e2b5972" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 30 18:59:32.519885 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:32.519805 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:32.604602 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:32.604567 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-03362d-6fdbdf99fb-72dp5" event={"ID":"efd98aa3-a350-4bf9-82ec-34f7b7917adc","Type":"ContainerStarted","Data":"fb1bc614e60821470b40ad5260ca11e70a89f7cfce208380e89642a08aeaeb70"} Oct 30 18:59:32.604887 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:32.604870 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-03362d-6fdbdf99fb-72dp5" Oct 30 18:59:32.623206 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:32.621335 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-03362d-6fdbdf99fb-72dp5" podStartSLOduration=2.97033012 podStartE2EDuration="3.621321477s" totalImagesPullingTime="650.991357ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:59:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:59:32.617813309 +0000 UTC m=+1540.146844447" watchObservedRunningTime="2026-10-30 18:59:32.621321477 +0000 UTC m=+1540.150352626" Oct 30 18:59:32.804692 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:32.801221 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c3a5b9-787455d59f-kd9ng"] Oct 30 18:59:32.814508 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:32.814478 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c3a5b9-787455d59f-kd9ng"] Oct 30 18:59:32.814887 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:32.814800 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c3a5b9-787455d59f-kd9ng" Oct 30 18:59:32.838334 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:32.836701 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-c3a5b9-serving-cert\"" Oct 30 18:59:32.902166 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:32.895413 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:32.902166 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:32.896558 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a59257f8-0671-461d-ac61-4956961be8f8-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-c3a5b9-787455d59f-kd9ng\" (UID: \"a59257f8-0671-461d-ac61-4956961be8f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c3a5b9-787455d59f-kd9ng" Oct 30 18:59:32.902166 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:32.896719 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a59257f8-0671-461d-ac61-4956961be8f8-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-c3a5b9-787455d59f-kd9ng\" (UID: \"a59257f8-0671-461d-ac61-4956961be8f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c3a5b9-787455d59f-kd9ng" Oct 30 18:59:32.902166 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:32.896799 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a59257f8-0671-461d-ac61-4956961be8f8-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-c3a5b9-787455d59f-kd9ng\" (UID: \"a59257f8-0671-461d-ac61-4956961be8f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c3a5b9-787455d59f-kd9ng" Oct 30 18:59:32.902166 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:32.896820 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swbbj\" (UniqueName: \"kubernetes.io/projected/a59257f8-0671-461d-ac61-4956961be8f8-kube-api-access-swbbj\") pod \"maas-api-e2e-shared-a-w1-c3a5b9-787455d59f-kd9ng\" (UID: \"a59257f8-0671-461d-ac61-4956961be8f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c3a5b9-787455d59f-kd9ng" Oct 30 18:59:33.011224 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:33.007076 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a59257f8-0671-461d-ac61-4956961be8f8-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-c3a5b9-787455d59f-kd9ng\" (UID: \"a59257f8-0671-461d-ac61-4956961be8f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c3a5b9-787455d59f-kd9ng" Oct 30 18:59:33.011224 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:33.007131 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swbbj\" (UniqueName: \"kubernetes.io/projected/a59257f8-0671-461d-ac61-4956961be8f8-kube-api-access-swbbj\") pod \"maas-api-e2e-shared-a-w1-c3a5b9-787455d59f-kd9ng\" (UID: \"a59257f8-0671-461d-ac61-4956961be8f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c3a5b9-787455d59f-kd9ng" Oct 30 18:59:33.011224 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:33.007229 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a59257f8-0671-461d-ac61-4956961be8f8-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-c3a5b9-787455d59f-kd9ng\" (UID: \"a59257f8-0671-461d-ac61-4956961be8f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c3a5b9-787455d59f-kd9ng" Oct 30 18:59:33.011224 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:33.007434 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a59257f8-0671-461d-ac61-4956961be8f8-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-c3a5b9-787455d59f-kd9ng\" (UID: \"a59257f8-0671-461d-ac61-4956961be8f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c3a5b9-787455d59f-kd9ng" Oct 30 18:59:33.011224 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:33.008282 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a59257f8-0671-461d-ac61-4956961be8f8-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-c3a5b9-787455d59f-kd9ng\" (UID: \"a59257f8-0671-461d-ac61-4956961be8f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c3a5b9-787455d59f-kd9ng" Oct 30 18:59:33.011224 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:59:33.008395 6960 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c3a5b9-serving-cert: secret "maas-api-e2e-shared-a-w1-c3a5b9-serving-cert" not found Oct 30 18:59:33.011224 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:59:33.008456 6960 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a59257f8-0671-461d-ac61-4956961be8f8-maas-api-tls podName:a59257f8-0671-461d-ac61-4956961be8f8 nodeName:}" failed. No retries permitted until 2026-10-30 18:59:33.508436012 +0000 UTC m=+1541.037467145 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a59257f8-0671-461d-ac61-4956961be8f8-maas-api-tls") pod "maas-api-e2e-shared-a-w1-c3a5b9-787455d59f-kd9ng" (UID: "a59257f8-0671-461d-ac61-4956961be8f8") : secret "maas-api-e2e-shared-a-w1-c3a5b9-serving-cert" not found Oct 30 18:59:33.024811 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:33.024778 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a59257f8-0671-461d-ac61-4956961be8f8-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-c3a5b9-787455d59f-kd9ng\" (UID: \"a59257f8-0671-461d-ac61-4956961be8f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c3a5b9-787455d59f-kd9ng" Oct 30 18:59:33.035181 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:33.035128 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:33.047982 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:33.047540 6960 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-gtlvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 30 18:59:33.047982 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:33.047601 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-gtlvx" podUID="2c4cb031-1d4b-4026-8846-bcf16e2b5972" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 30 18:59:33.061232 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:33.058304 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swbbj\" (UniqueName: \"kubernetes.io/projected/a59257f8-0671-461d-ac61-4956961be8f8-kube-api-access-swbbj\") pod \"maas-api-e2e-shared-a-w1-c3a5b9-787455d59f-kd9ng\" (UID: \"a59257f8-0671-461d-ac61-4956961be8f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c3a5b9-787455d59f-kd9ng" Oct 30 18:59:33.518261 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:33.517343 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a59257f8-0671-461d-ac61-4956961be8f8-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-c3a5b9-787455d59f-kd9ng\" (UID: \"a59257f8-0671-461d-ac61-4956961be8f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c3a5b9-787455d59f-kd9ng" Oct 30 18:59:33.518261 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:59:33.517565 6960 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c3a5b9-serving-cert: secret "maas-api-e2e-shared-a-w1-c3a5b9-serving-cert" not found Oct 30 18:59:33.518261 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:59:33.517628 6960 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a59257f8-0671-461d-ac61-4956961be8f8-maas-api-tls podName:a59257f8-0671-461d-ac61-4956961be8f8 nodeName:}" failed. No retries permitted until 2026-10-30 18:59:34.517611036 +0000 UTC m=+1542.046642167 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a59257f8-0671-461d-ac61-4956961be8f8-maas-api-tls") pod "maas-api-e2e-shared-a-w1-c3a5b9-787455d59f-kd9ng" (UID: "a59257f8-0671-461d-ac61-4956961be8f8") : secret "maas-api-e2e-shared-a-w1-c3a5b9-serving-cert" not found Oct 30 18:59:33.703918 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:33.703882 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:33.904175 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:33.899418 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:34.043168 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:34.042860 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:34.054516 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:34.049526 6960 patch_prober.go:29] interesting pod/e2e-worker-w1-aa6736-openshift-default-6f74f55675-gtlvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 30 18:59:34.054516 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:34.049599 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-gtlvx" podUID="2c4cb031-1d4b-4026-8846-bcf16e2b5972" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 30 18:59:34.176757 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:34.176665 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:34.289462 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:34.289429 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:34.558431 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:34.554048 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a59257f8-0671-461d-ac61-4956961be8f8-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-c3a5b9-787455d59f-kd9ng\" (UID: \"a59257f8-0671-461d-ac61-4956961be8f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c3a5b9-787455d59f-kd9ng" Oct 30 18:59:34.564170 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:34.560215 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a59257f8-0671-461d-ac61-4956961be8f8-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-c3a5b9-787455d59f-kd9ng\" (UID: \"a59257f8-0671-461d-ac61-4956961be8f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c3a5b9-787455d59f-kd9ng" Oct 30 18:59:34.597558 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:34.597520 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:34.652576 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:34.652546 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c3a5b9-787455d59f-kd9ng" Oct 30 18:59:34.966467 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:34.962953 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c3a5b9-787455d59f-kd9ng"] Oct 30 18:59:35.376703 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:35.375738 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-gtlvx" Oct 30 18:59:35.377235 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:35.377213 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-gtlvx" Oct 30 18:59:35.377963 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:35.377943 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-gtlvx" Oct 30 18:59:35.463181 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:35.462691 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:35.626521 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:35.626429 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c3a5b9-787455d59f-kd9ng" event={"ID":"a59257f8-0671-461d-ac61-4956961be8f8","Type":"ContainerStarted","Data":"0ba38578db9019b742e5403c4b2f0a71ef0545854a6b5b2f1ca2d0cc3c4034e1"} Oct 30 18:59:35.626521 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:35.626494 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c3a5b9-787455d59f-kd9ng" event={"ID":"a59257f8-0671-461d-ac61-4956961be8f8","Type":"ContainerStarted","Data":"4da81e17bd88a25e0df4805357c84430c9c124a25f0c89114bc4876713c768af"} Oct 30 18:59:35.626746 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:35.626705 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c3a5b9-787455d59f-kd9ng" Oct 30 18:59:35.664339 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:35.664293 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c3a5b9-787455d59f-kd9ng" podStartSLOduration=3.264011963 podStartE2EDuration="3.664278995s" totalImagesPullingTime="400.267032ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:59:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:59:35.661877638 +0000 UTC m=+1543.190908777" watchObservedRunningTime="2026-10-30 18:59:35.664278995 +0000 UTC m=+1543.193310133" Oct 30 18:59:36.478993 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:36.478952 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:37.294637 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:37.294598 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:38.308921 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:38.308859 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:39.122516 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:39.122474 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:39.247694 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:39.247647 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:39.619659 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:39.619631 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-03362d-6fdbdf99fb-72dp5" Oct 30 18:59:39.637300 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:39.637256 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:40.402019 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:40.401986 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:40.557912 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:40.557876 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:40.840204 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:40.840170 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:41.643005 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:41.642976 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c3a5b9-787455d59f-kd9ng" Oct 30 18:59:42.009740 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:42.009633 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:42.603002 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:42.602962 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:42.736344 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:42.736259 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:42.992819 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:42.992736 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:43.174171 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:43.173135 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:43.996752 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:43.996711 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:45.087099 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:45.087060 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:46.043382 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:46.043349 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:47.163389 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:47.163338 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:47.857640 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:47.857605 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:47.971649 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:47.971614 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:48.095370 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:48.095332 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:48.449869 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:48.449829 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:48.558022 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:48.557991 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:48.887749 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:48.887716 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:49.051992 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:49.051957 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:49.226859 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:49.226715 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:49.816131 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:49.816083 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:50.649905 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:50.649872 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:50.999320 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:50.999236 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:51.149585 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:51.149548 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:51.278889 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:51.278799 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:51.700619 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:51.700585 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:51.848168 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:51.837650 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:52.182740 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:52.182709 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:52.549217 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:52.548476 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:52.616771 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:52.616736 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-86c150/auth-test-model-86c150-kserve-84bf95dfd5-5qc64"] Oct 30 18:59:52.621464 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:52.621443 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-86c150/auth-test-model-86c150-kserve-84bf95dfd5-5qc64"] Oct 30 18:59:52.622006 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:52.621987 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-86c150/auth-test-model-86c150-kserve-84bf95dfd5-5qc64" Oct 30 18:59:52.631410 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:52.631335 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-86c150\"/\"default-dockercfg-m7v4z\"" Oct 30 18:59:52.631496 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:52.631441 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-86c150\"/\"kube-root-ca.crt\"" Oct 30 18:59:52.631564 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:52.631542 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-86c150\"/\"openshift-service-ca.crt\"" Oct 30 18:59:52.632133 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:52.631866 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-86c150\"/\"auth-test-model-86c150-kserve-self-signed-certs\"" Oct 30 18:59:52.695704 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:52.695668 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:52.798254 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:52.798216 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ee03adc-306c-4acb-862b-929b8433b6ac-home\") pod \"auth-test-model-86c150-kserve-84bf95dfd5-5qc64\" (UID: \"7ee03adc-306c-4acb-862b-929b8433b6ac\") " pod="ai-tenant-e2e-shared-a-w4-86c150/auth-test-model-86c150-kserve-84bf95dfd5-5qc64" Oct 30 18:59:52.798254 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:52.798248 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ee03adc-306c-4acb-862b-929b8433b6ac-dshm\") pod \"auth-test-model-86c150-kserve-84bf95dfd5-5qc64\" (UID: \"7ee03adc-306c-4acb-862b-929b8433b6ac\") " pod="ai-tenant-e2e-shared-a-w4-86c150/auth-test-model-86c150-kserve-84bf95dfd5-5qc64" Oct 30 18:59:52.798527 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:52.798431 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ee03adc-306c-4acb-862b-929b8433b6ac-tmp-dir\") pod \"auth-test-model-86c150-kserve-84bf95dfd5-5qc64\" (UID: \"7ee03adc-306c-4acb-862b-929b8433b6ac\") " pod="ai-tenant-e2e-shared-a-w4-86c150/auth-test-model-86c150-kserve-84bf95dfd5-5qc64" Oct 30 18:59:52.798589 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:52.798559 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ee03adc-306c-4acb-862b-929b8433b6ac-model-cache\") pod \"auth-test-model-86c150-kserve-84bf95dfd5-5qc64\" (UID: \"7ee03adc-306c-4acb-862b-929b8433b6ac\") " pod="ai-tenant-e2e-shared-a-w4-86c150/auth-test-model-86c150-kserve-84bf95dfd5-5qc64" Oct 30 18:59:52.798667 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:52.798646 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ee03adc-306c-4acb-862b-929b8433b6ac-tls-certs\") pod \"auth-test-model-86c150-kserve-84bf95dfd5-5qc64\" (UID: \"7ee03adc-306c-4acb-862b-929b8433b6ac\") " pod="ai-tenant-e2e-shared-a-w4-86c150/auth-test-model-86c150-kserve-84bf95dfd5-5qc64" Oct 30 18:59:52.798872 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:52.798847 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8lnt\" (UniqueName: \"kubernetes.io/projected/7ee03adc-306c-4acb-862b-929b8433b6ac-kube-api-access-v8lnt\") pod \"auth-test-model-86c150-kserve-84bf95dfd5-5qc64\" (UID: \"7ee03adc-306c-4acb-862b-929b8433b6ac\") " pod="ai-tenant-e2e-shared-a-w4-86c150/auth-test-model-86c150-kserve-84bf95dfd5-5qc64" Oct 30 18:59:52.900520 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:52.900486 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8lnt\" (UniqueName: \"kubernetes.io/projected/7ee03adc-306c-4acb-862b-929b8433b6ac-kube-api-access-v8lnt\") pod \"auth-test-model-86c150-kserve-84bf95dfd5-5qc64\" (UID: \"7ee03adc-306c-4acb-862b-929b8433b6ac\") " pod="ai-tenant-e2e-shared-a-w4-86c150/auth-test-model-86c150-kserve-84bf95dfd5-5qc64" Oct 30 18:59:52.901283 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:52.901255 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ee03adc-306c-4acb-862b-929b8433b6ac-home\") pod \"auth-test-model-86c150-kserve-84bf95dfd5-5qc64\" (UID: \"7ee03adc-306c-4acb-862b-929b8433b6ac\") " pod="ai-tenant-e2e-shared-a-w4-86c150/auth-test-model-86c150-kserve-84bf95dfd5-5qc64" Oct 30 18:59:52.901883 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:52.901864 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ee03adc-306c-4acb-862b-929b8433b6ac-dshm\") pod \"auth-test-model-86c150-kserve-84bf95dfd5-5qc64\" (UID: \"7ee03adc-306c-4acb-862b-929b8433b6ac\") " pod="ai-tenant-e2e-shared-a-w4-86c150/auth-test-model-86c150-kserve-84bf95dfd5-5qc64" Oct 30 18:59:52.902522 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:52.902504 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ee03adc-306c-4acb-862b-929b8433b6ac-tmp-dir\") pod \"auth-test-model-86c150-kserve-84bf95dfd5-5qc64\" (UID: \"7ee03adc-306c-4acb-862b-929b8433b6ac\") " pod="ai-tenant-e2e-shared-a-w4-86c150/auth-test-model-86c150-kserve-84bf95dfd5-5qc64" Oct 30 18:59:52.903000 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:52.902982 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ee03adc-306c-4acb-862b-929b8433b6ac-model-cache\") pod \"auth-test-model-86c150-kserve-84bf95dfd5-5qc64\" (UID: \"7ee03adc-306c-4acb-862b-929b8433b6ac\") " pod="ai-tenant-e2e-shared-a-w4-86c150/auth-test-model-86c150-kserve-84bf95dfd5-5qc64" Oct 30 18:59:52.903208 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:52.903190 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ee03adc-306c-4acb-862b-929b8433b6ac-tls-certs\") pod \"auth-test-model-86c150-kserve-84bf95dfd5-5qc64\" (UID: \"7ee03adc-306c-4acb-862b-929b8433b6ac\") " pod="ai-tenant-e2e-shared-a-w4-86c150/auth-test-model-86c150-kserve-84bf95dfd5-5qc64" Oct 30 18:59:52.903954 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:52.903935 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ee03adc-306c-4acb-862b-929b8433b6ac-model-cache\") pod \"auth-test-model-86c150-kserve-84bf95dfd5-5qc64\" (UID: \"7ee03adc-306c-4acb-862b-929b8433b6ac\") " pod="ai-tenant-e2e-shared-a-w4-86c150/auth-test-model-86c150-kserve-84bf95dfd5-5qc64" Oct 30 18:59:52.904082 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:52.901811 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ee03adc-306c-4acb-862b-929b8433b6ac-home\") pod \"auth-test-model-86c150-kserve-84bf95dfd5-5qc64\" (UID: \"7ee03adc-306c-4acb-862b-929b8433b6ac\") " pod="ai-tenant-e2e-shared-a-w4-86c150/auth-test-model-86c150-kserve-84bf95dfd5-5qc64" Oct 30 18:59:52.933606 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:52.902895 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ee03adc-306c-4acb-862b-929b8433b6ac-tmp-dir\") pod \"auth-test-model-86c150-kserve-84bf95dfd5-5qc64\" (UID: \"7ee03adc-306c-4acb-862b-929b8433b6ac\") " pod="ai-tenant-e2e-shared-a-w4-86c150/auth-test-model-86c150-kserve-84bf95dfd5-5qc64" Oct 30 18:59:52.933606 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:52.915648 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ee03adc-306c-4acb-862b-929b8433b6ac-dshm\") pod \"auth-test-model-86c150-kserve-84bf95dfd5-5qc64\" (UID: \"7ee03adc-306c-4acb-862b-929b8433b6ac\") " pod="ai-tenant-e2e-shared-a-w4-86c150/auth-test-model-86c150-kserve-84bf95dfd5-5qc64" Oct 30 18:59:52.933606 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:52.916038 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ee03adc-306c-4acb-862b-929b8433b6ac-tls-certs\") pod \"auth-test-model-86c150-kserve-84bf95dfd5-5qc64\" (UID: \"7ee03adc-306c-4acb-862b-929b8433b6ac\") " pod="ai-tenant-e2e-shared-a-w4-86c150/auth-test-model-86c150-kserve-84bf95dfd5-5qc64" Oct 30 18:59:52.933606 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:52.927613 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8lnt\" (UniqueName: \"kubernetes.io/projected/7ee03adc-306c-4acb-862b-929b8433b6ac-kube-api-access-v8lnt\") pod \"auth-test-model-86c150-kserve-84bf95dfd5-5qc64\" (UID: \"7ee03adc-306c-4acb-862b-929b8433b6ac\") " pod="ai-tenant-e2e-shared-a-w4-86c150/auth-test-model-86c150-kserve-84bf95dfd5-5qc64" Oct 30 18:59:52.946794 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:52.943448 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-86c150/auth-test-model-86c150-kserve-84bf95dfd5-5qc64" Oct 30 18:59:53.001173 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:53.000648 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:53.124209 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:53.122830 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:53.333441 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:53.333411 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-86c150/auth-test-model-86c150-kserve-84bf95dfd5-5qc64"] Oct 30 18:59:53.334859 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:59:53.334829 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7ee03adc_306c_4acb_862b_929b8433b6ac.slice/crio-902801b2ef3d1ff26ffe482d4cc1acf701307d2623203250456d31ba9e888c7f WatchSource:0}: Error finding container 902801b2ef3d1ff26ffe482d4cc1acf701307d2623203250456d31ba9e888c7f: Status 404 returned error can't find the container with id 902801b2ef3d1ff26ffe482d4cc1acf701307d2623203250456d31ba9e888c7f Oct 30 18:59:53.835850 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:53.835820 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-86c150/auth-test-model-86c150-kserve-84bf95dfd5-5qc64" event={"ID":"7ee03adc-306c-4acb-862b-929b8433b6ac","Type":"ContainerStarted","Data":"e8c27f55957af342da8da48a9a54ac64c76a3ea469a0d3adf600eb22b9237233"} Oct 30 18:59:53.836027 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:53.835862 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-86c150/auth-test-model-86c150-kserve-84bf95dfd5-5qc64" event={"ID":"7ee03adc-306c-4acb-862b-929b8433b6ac","Type":"ContainerStarted","Data":"902801b2ef3d1ff26ffe482d4cc1acf701307d2623203250456d31ba9e888c7f"} Oct 30 18:59:53.860087 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:53.860028 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-86c150/auth-test-model-86c150-kserve-84bf95dfd5-5qc64" podStartSLOduration=1.860008268 podStartE2EDuration="1.860008268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:59:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:59:53.855492074 +0000 UTC m=+1561.384523213" watchObservedRunningTime="2026-10-30 18:59:53.860008268 +0000 UTC m=+1561.389039408" Oct 30 18:59:53.913117 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:53.913066 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:53.934689 ip-10-0-140-163 kubenswrapper[6960]: E1030 18:59:53.934640 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 18:59:53.948031 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:53.948006 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-86c150/auth-test-model-86c150-kserve-84bf95dfd5-5qc64" Oct 30 18:59:54.260996 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:54.260913 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:54.310007 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:54.309975 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-743e0b-85c6498fc5-hdpjx"] Oct 30 18:59:54.314372 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:54.314353 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-743e0b-85c6498fc5-hdpjx" Oct 30 18:59:54.325218 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:54.321757 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-743e0b\"" Oct 30 18:59:54.325218 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:54.322426 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-743e0b-dockercfg-8g6ps\"" Oct 30 18:59:54.326208 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:54.326184 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-743e0b-85c6498fc5-hdpjx"] Oct 30 18:59:54.351543 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:54.351508 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b1ff70c-fcdf-4cdc-904c-37cd17e95ded-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-743e0b-85c6498fc5-hdpjx\" (UID: \"4b1ff70c-fcdf-4cdc-904c-37cd17e95ded\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-743e0b-85c6498fc5-hdpjx" Oct 30 18:59:54.351985 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:54.351555 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4ltx\" (UniqueName: \"kubernetes.io/projected/4b1ff70c-fcdf-4cdc-904c-37cd17e95ded-kube-api-access-b4ltx\") pod \"payload-pre-processing-e2e-explicit-743e0b-85c6498fc5-hdpjx\" (UID: \"4b1ff70c-fcdf-4cdc-904c-37cd17e95ded\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-743e0b-85c6498fc5-hdpjx" Oct 30 18:59:54.431167 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:54.429320 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:54.477169 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:54.467208 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b1ff70c-fcdf-4cdc-904c-37cd17e95ded-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-743e0b-85c6498fc5-hdpjx\" (UID: \"4b1ff70c-fcdf-4cdc-904c-37cd17e95ded\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-743e0b-85c6498fc5-hdpjx" Oct 30 18:59:54.477169 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:54.467259 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4ltx\" (UniqueName: \"kubernetes.io/projected/4b1ff70c-fcdf-4cdc-904c-37cd17e95ded-kube-api-access-b4ltx\") pod \"payload-pre-processing-e2e-explicit-743e0b-85c6498fc5-hdpjx\" (UID: \"4b1ff70c-fcdf-4cdc-904c-37cd17e95ded\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-743e0b-85c6498fc5-hdpjx" Oct 30 18:59:54.477169 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:54.468518 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b1ff70c-fcdf-4cdc-904c-37cd17e95ded-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-743e0b-85c6498fc5-hdpjx\" (UID: \"4b1ff70c-fcdf-4cdc-904c-37cd17e95ded\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-743e0b-85c6498fc5-hdpjx" Oct 30 18:59:54.509579 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:54.509496 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4ltx\" (UniqueName: \"kubernetes.io/projected/4b1ff70c-fcdf-4cdc-904c-37cd17e95ded-kube-api-access-b4ltx\") pod \"payload-pre-processing-e2e-explicit-743e0b-85c6498fc5-hdpjx\" (UID: \"4b1ff70c-fcdf-4cdc-904c-37cd17e95ded\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-743e0b-85c6498fc5-hdpjx" Oct 30 18:59:54.631370 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:54.631340 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-743e0b-85c6498fc5-hdpjx" Oct 30 18:59:54.919585 ip-10-0-140-163 kubenswrapper[6960]: W1030 18:59:54.919556 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b1ff70c_fcdf_4cdc_904c_37cd17e95ded.slice/crio-ca257970f8026dbb8b5b3e3e08bfa44c43ba4205ff5681df5400ea4e679ef2e4 WatchSource:0}: Error finding container ca257970f8026dbb8b5b3e3e08bfa44c43ba4205ff5681df5400ea4e679ef2e4: Status 404 returned error can't find the container with id ca257970f8026dbb8b5b3e3e08bfa44c43ba4205ff5681df5400ea4e679ef2e4 Oct 30 18:59:54.920885 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:54.920863 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-743e0b-85c6498fc5-hdpjx"] Oct 30 18:59:55.231467 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:55.231358 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:55.852099 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:55.852064 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-743e0b-85c6498fc5-hdpjx" event={"ID":"4b1ff70c-fcdf-4cdc-904c-37cd17e95ded","Type":"ContainerStarted","Data":"05355ffe6184f3946e526eb1fc932f2c3b6a5648acc42bb6e7aa1a059324f027"} Oct 30 18:59:55.852504 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:55.852126 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-743e0b-85c6498fc5-hdpjx" event={"ID":"4b1ff70c-fcdf-4cdc-904c-37cd17e95ded","Type":"ContainerStarted","Data":"ca257970f8026dbb8b5b3e3e08bfa44c43ba4205ff5681df5400ea4e679ef2e4"} Oct 30 18:59:55.888967 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:55.888912 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-743e0b-85c6498fc5-hdpjx" podStartSLOduration=1.888898963 podStartE2EDuration="1.888898963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:59:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:59:55.886122276 +0000 UTC m=+1563.415153429" watchObservedRunningTime="2026-10-30 18:59:55.888898963 +0000 UTC m=+1563.417930100" Oct 30 18:59:55.936584 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:55.936528 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:56.108696 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:56.108595 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:56.384059 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:56.383955 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:56.860743 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:56.860687 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-743e0b-85c6498fc5-hdpjx" Oct 30 18:59:57.249785 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:57.249702 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:57.415722 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:57.415685 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:57.801063 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:57.801029 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:59.073802 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:59.073761 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:59.889594 ip-10-0-140-163 kubenswrapper[6960]: I1030 18:59:59.889547 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:00.195901 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:00.195822 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-03362d-29889780-hwd5x"] Oct 30 19:00:00.201903 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:00.201866 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-03362d-29889780-hwd5x" Oct 30 19:00:00.202834 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:00.202812 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da70a5-29889780-dn54w"] Oct 30 19:00:00.214410 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:00.214361 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da70a5-29889780-dn54w" Oct 30 19:00:00.217918 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:00.217890 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-03362d-29889780-hwd5x"] Oct 30 19:00:00.232951 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:00.232926 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da70a5-29889780-dn54w"] Oct 30 19:00:00.238111 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:00.238084 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ab7df0-29889780-z66fz"] Oct 30 19:00:00.244772 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:00.244749 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ab7df0-29889780-z66fz" Oct 30 19:00:00.255075 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:00.255045 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ab7df0-29889780-z66fz"] Oct 30 19:00:00.266361 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:00.266111 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\"" Oct 30 19:00:00.273615 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:00.272109 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889780-qmbj9"] Oct 30 19:00:00.277279 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:00.277255 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889780-qmbj9" Oct 30 19:00:00.283197 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:00.283167 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba9e1a42-4e4d-4b4a-8a2b-28823c69e663-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-03362d-29889780-hwd5x\" (UID: \"ba9e1a42-4e4d-4b4a-8a2b-28823c69e663\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-03362d-29889780-hwd5x" Oct 30 19:00:00.283464 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:00.283442 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4snc\" (UniqueName: \"kubernetes.io/projected/734520a1-484d-4cc4-a340-302045e8ab82-kube-api-access-w4snc\") pod \"maas-api-key-cleanup-e2e-worker-w4-da70a5-29889780-dn54w\" (UID: \"734520a1-484d-4cc4-a340-302045e8ab82\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da70a5-29889780-dn54w" Oct 30 19:00:00.283669 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:00.283649 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfgpb\" (UniqueName: \"kubernetes.io/projected/0053dbc6-5623-498a-9fcf-9e89e1dc18ee-kube-api-access-bfgpb\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-ab7df0-29889780-z66fz\" (UID: \"0053dbc6-5623-498a-9fcf-9e89e1dc18ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ab7df0-29889780-z66fz" Oct 30 19:00:00.284568 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:00.284545 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvh5d\" (UniqueName: \"kubernetes.io/projected/ba9e1a42-4e4d-4b4a-8a2b-28823c69e663-kube-api-access-vvh5d\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-03362d-29889780-hwd5x\" (UID: \"ba9e1a42-4e4d-4b4a-8a2b-28823c69e663\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-03362d-29889780-hwd5x" Oct 30 19:00:00.284871 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:00.284845 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0053dbc6-5623-498a-9fcf-9e89e1dc18ee-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-ab7df0-29889780-z66fz\" (UID: \"0053dbc6-5623-498a-9fcf-9e89e1dc18ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ab7df0-29889780-z66fz" Oct 30 19:00:00.284959 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:00.284899 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/734520a1-484d-4cc4-a340-302045e8ab82-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-da70a5-29889780-dn54w\" (UID: \"734520a1-484d-4cc4-a340-302045e8ab82\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da70a5-29889780-dn54w" Oct 30 19:00:00.291562 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:00.291072 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889780-qmbj9"] Oct 30 19:00:00.386663 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:00.386620 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvh5d\" (UniqueName: \"kubernetes.io/projected/ba9e1a42-4e4d-4b4a-8a2b-28823c69e663-kube-api-access-vvh5d\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-03362d-29889780-hwd5x\" (UID: \"ba9e1a42-4e4d-4b4a-8a2b-28823c69e663\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-03362d-29889780-hwd5x" Oct 30 19:00:00.386879 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:00.386793 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0053dbc6-5623-498a-9fcf-9e89e1dc18ee-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-ab7df0-29889780-z66fz\" (UID: \"0053dbc6-5623-498a-9fcf-9e89e1dc18ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ab7df0-29889780-z66fz" Oct 30 19:00:00.386879 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:00.386840 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/734520a1-484d-4cc4-a340-302045e8ab82-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-da70a5-29889780-dn54w\" (UID: \"734520a1-484d-4cc4-a340-302045e8ab82\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da70a5-29889780-dn54w" Oct 30 19:00:00.387014 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:00.386876 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vs6n4\" (UniqueName: \"kubernetes.io/projected/ef261723-21fe-493d-940f-394b9aca8fbc-kube-api-access-vs6n4\") pod \"maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889780-qmbj9\" (UID: \"ef261723-21fe-493d-940f-394b9aca8fbc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889780-qmbj9" Oct 30 19:00:00.387014 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:00.386968 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba9e1a42-4e4d-4b4a-8a2b-28823c69e663-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-03362d-29889780-hwd5x\" (UID: \"ba9e1a42-4e4d-4b4a-8a2b-28823c69e663\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-03362d-29889780-hwd5x" Oct 30 19:00:00.387116 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:00.387091 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4snc\" (UniqueName: \"kubernetes.io/projected/734520a1-484d-4cc4-a340-302045e8ab82-kube-api-access-w4snc\") pod \"maas-api-key-cleanup-e2e-worker-w4-da70a5-29889780-dn54w\" (UID: \"734520a1-484d-4cc4-a340-302045e8ab82\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da70a5-29889780-dn54w" Oct 30 19:00:00.387214 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:00.387192 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef261723-21fe-493d-940f-394b9aca8fbc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889780-qmbj9\" (UID: \"ef261723-21fe-493d-940f-394b9aca8fbc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889780-qmbj9" Oct 30 19:00:00.387268 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:00.387251 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfgpb\" (UniqueName: \"kubernetes.io/projected/0053dbc6-5623-498a-9fcf-9e89e1dc18ee-kube-api-access-bfgpb\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-ab7df0-29889780-z66fz\" (UID: \"0053dbc6-5623-498a-9fcf-9e89e1dc18ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ab7df0-29889780-z66fz" Oct 30 19:00:00.387579 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:00.387562 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0053dbc6-5623-498a-9fcf-9e89e1dc18ee-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-ab7df0-29889780-z66fz\" (UID: \"0053dbc6-5623-498a-9fcf-9e89e1dc18ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ab7df0-29889780-z66fz" Oct 30 19:00:00.387618 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:00.387590 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba9e1a42-4e4d-4b4a-8a2b-28823c69e663-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-03362d-29889780-hwd5x\" (UID: \"ba9e1a42-4e4d-4b4a-8a2b-28823c69e663\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-03362d-29889780-hwd5x" Oct 30 19:00:00.387655 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:00.387635 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/734520a1-484d-4cc4-a340-302045e8ab82-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-da70a5-29889780-dn54w\" (UID: \"734520a1-484d-4cc4-a340-302045e8ab82\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da70a5-29889780-dn54w" Oct 30 19:00:00.403716 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:00.403670 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfgpb\" (UniqueName: \"kubernetes.io/projected/0053dbc6-5623-498a-9fcf-9e89e1dc18ee-kube-api-access-bfgpb\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-ab7df0-29889780-z66fz\" (UID: \"0053dbc6-5623-498a-9fcf-9e89e1dc18ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ab7df0-29889780-z66fz" Oct 30 19:00:00.403886 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:00.403771 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4snc\" (UniqueName: \"kubernetes.io/projected/734520a1-484d-4cc4-a340-302045e8ab82-kube-api-access-w4snc\") pod \"maas-api-key-cleanup-e2e-worker-w4-da70a5-29889780-dn54w\" (UID: \"734520a1-484d-4cc4-a340-302045e8ab82\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da70a5-29889780-dn54w" Oct 30 19:00:00.404133 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:00.404107 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvh5d\" (UniqueName: \"kubernetes.io/projected/ba9e1a42-4e4d-4b4a-8a2b-28823c69e663-kube-api-access-vvh5d\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-03362d-29889780-hwd5x\" (UID: \"ba9e1a42-4e4d-4b4a-8a2b-28823c69e663\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-03362d-29889780-hwd5x" Oct 30 19:00:00.489202 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:00.489065 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vs6n4\" (UniqueName: \"kubernetes.io/projected/ef261723-21fe-493d-940f-394b9aca8fbc-kube-api-access-vs6n4\") pod \"maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889780-qmbj9\" (UID: \"ef261723-21fe-493d-940f-394b9aca8fbc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889780-qmbj9" Oct 30 19:00:00.489364 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:00.489294 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef261723-21fe-493d-940f-394b9aca8fbc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889780-qmbj9\" (UID: \"ef261723-21fe-493d-940f-394b9aca8fbc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889780-qmbj9" Oct 30 19:00:00.490003 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:00.489983 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef261723-21fe-493d-940f-394b9aca8fbc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889780-qmbj9\" (UID: \"ef261723-21fe-493d-940f-394b9aca8fbc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889780-qmbj9" Oct 30 19:00:00.502879 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:00.502845 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vs6n4\" (UniqueName: \"kubernetes.io/projected/ef261723-21fe-493d-940f-394b9aca8fbc-kube-api-access-vs6n4\") pod \"maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889780-qmbj9\" (UID: \"ef261723-21fe-493d-940f-394b9aca8fbc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889780-qmbj9" Oct 30 19:00:00.540692 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:00.540659 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-03362d-29889780-hwd5x" Oct 30 19:00:00.570215 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:00.570185 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da70a5-29889780-dn54w" Oct 30 19:00:00.580818 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:00.580790 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ab7df0-29889780-z66fz" Oct 30 19:00:00.606103 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:00.602935 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889780-qmbj9" Oct 30 19:00:00.835751 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:00.835730 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-03362d-29889780-hwd5x"] Oct 30 19:00:00.838256 ip-10-0-140-163 kubenswrapper[6960]: W1030 19:00:00.838224 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba9e1a42_4e4d_4b4a_8a2b_28823c69e663.slice/crio-3a3ac0947f0ff4d83d22110adcee7a9051377b906fe9d94f564fadcaf6e3917a WatchSource:0}: Error finding container 3a3ac0947f0ff4d83d22110adcee7a9051377b906fe9d94f564fadcaf6e3917a: Status 404 returned error can't find the container with id 3a3ac0947f0ff4d83d22110adcee7a9051377b906fe9d94f564fadcaf6e3917a Oct 30 19:00:00.883824 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:00.883799 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da70a5-29889780-dn54w"] Oct 30 19:00:00.884464 ip-10-0-140-163 kubenswrapper[6960]: W1030 19:00:00.884437 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod734520a1_484d_4cc4_a340_302045e8ab82.slice/crio-106de254aa0a48248411234e80567380b2bf5cc4e883fb5512fb5535846e34f0 WatchSource:0}: Error finding container 106de254aa0a48248411234e80567380b2bf5cc4e883fb5512fb5535846e34f0: Status 404 returned error can't find the container with id 106de254aa0a48248411234e80567380b2bf5cc4e883fb5512fb5535846e34f0 Oct 30 19:00:00.889076 ip-10-0-140-163 kubenswrapper[6960]: W1030 19:00:00.888983 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0053dbc6_5623_498a_9fcf_9e89e1dc18ee.slice/crio-f4ef1c57ccabd9975a6e823515ca4c86b99886f27fea8f9e5e5357148bf8a4a0 WatchSource:0}: Error finding container f4ef1c57ccabd9975a6e823515ca4c86b99886f27fea8f9e5e5357148bf8a4a0: Status 404 returned error can't find the container with id f4ef1c57ccabd9975a6e823515ca4c86b99886f27fea8f9e5e5357148bf8a4a0 Oct 30 19:00:00.890402 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:00.890377 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ab7df0-29889780-z66fz"] Oct 30 19:00:00.928492 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:00.928472 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889780-qmbj9"] Oct 30 19:00:00.929732 ip-10-0-140-163 kubenswrapper[6960]: W1030 19:00:00.929709 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef261723_21fe_493d_940f_394b9aca8fbc.slice/crio-249f41d0d4f5f96d2593baa800117c7774a5b2d30809b1ed5698763e555fdd0c WatchSource:0}: Error finding container 249f41d0d4f5f96d2593baa800117c7774a5b2d30809b1ed5698763e555fdd0c: Status 404 returned error can't find the container with id 249f41d0d4f5f96d2593baa800117c7774a5b2d30809b1ed5698763e555fdd0c Oct 30 19:00:00.965387 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:00.965326 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-03362d-29889780-hwd5x" event={"ID":"ba9e1a42-4e4d-4b4a-8a2b-28823c69e663","Type":"ContainerStarted","Data":"3a3ac0947f0ff4d83d22110adcee7a9051377b906fe9d94f564fadcaf6e3917a"} Oct 30 19:00:00.966592 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:00.966568 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da70a5-29889780-dn54w" event={"ID":"734520a1-484d-4cc4-a340-302045e8ab82","Type":"ContainerStarted","Data":"106de254aa0a48248411234e80567380b2bf5cc4e883fb5512fb5535846e34f0"} Oct 30 19:00:00.967930 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:00.967908 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ab7df0-29889780-z66fz" event={"ID":"0053dbc6-5623-498a-9fcf-9e89e1dc18ee","Type":"ContainerStarted","Data":"f4ef1c57ccabd9975a6e823515ca4c86b99886f27fea8f9e5e5357148bf8a4a0"} Oct 30 19:00:00.969348 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:00.969326 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889780-qmbj9" event={"ID":"ef261723-21fe-493d-940f-394b9aca8fbc","Type":"ContainerStarted","Data":"249f41d0d4f5f96d2593baa800117c7774a5b2d30809b1ed5698763e555fdd0c"} Oct 30 19:00:00.970464 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:00.970430 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:01.844564 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:01.844530 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:01.977572 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:01.977535 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-03362d-29889780-hwd5x" event={"ID":"ba9e1a42-4e4d-4b4a-8a2b-28823c69e663","Type":"ContainerStarted","Data":"0f65aaf8b82d3f251b47d7037cae09f017d5b179b0dd86b4f5d65cf7b0c481c0"} Oct 30 19:00:01.979574 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:01.979547 6960 generic.go:356] "Generic (PLEG): container finished" podID="734520a1-484d-4cc4-a340-302045e8ab82" containerID="0fd068d699d8997ef4d671a9a56ca423a17d260ee4d4192231a1e8e525fd7ba5" exitCode=60 Oct 30 19:00:01.979717 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:01.979628 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da70a5-29889780-dn54w" event={"ID":"734520a1-484d-4cc4-a340-302045e8ab82","Type":"ContainerDied","Data":"0fd068d699d8997ef4d671a9a56ca423a17d260ee4d4192231a1e8e525fd7ba5"} Oct 30 19:00:01.979892 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:01.979876 6960 scope.go:122] "RemoveContainer" containerID="0fd068d699d8997ef4d671a9a56ca423a17d260ee4d4192231a1e8e525fd7ba5" Oct 30 19:00:01.981958 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:01.981747 6960 generic.go:356] "Generic (PLEG): container finished" podID="0053dbc6-5623-498a-9fcf-9e89e1dc18ee" containerID="a203e30b15b62b40471f137b9afd553f91cb9324b23ad8af35846823d7e1ae25" exitCode=60 Oct 30 19:00:01.981958 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:01.981887 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ab7df0-29889780-z66fz" event={"ID":"0053dbc6-5623-498a-9fcf-9e89e1dc18ee","Type":"ContainerDied","Data":"a203e30b15b62b40471f137b9afd553f91cb9324b23ad8af35846823d7e1ae25"} Oct 30 19:00:01.982092 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:01.981997 6960 scope.go:122] "RemoveContainer" containerID="a203e30b15b62b40471f137b9afd553f91cb9324b23ad8af35846823d7e1ae25" Oct 30 19:00:01.983836 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:01.983816 6960 generic.go:356] "Generic (PLEG): container finished" podID="ef261723-21fe-493d-940f-394b9aca8fbc" containerID="06fdf63e4b70002d4a4e19427f9fd10342a9a57a878482c37b7656c5ae3ccc0f" exitCode=60 Oct 30 19:00:01.983924 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:01.983881 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889780-qmbj9" event={"ID":"ef261723-21fe-493d-940f-394b9aca8fbc","Type":"ContainerDied","Data":"06fdf63e4b70002d4a4e19427f9fd10342a9a57a878482c37b7656c5ae3ccc0f"} Oct 30 19:00:01.984052 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:01.984036 6960 scope.go:122] "RemoveContainer" containerID="06fdf63e4b70002d4a4e19427f9fd10342a9a57a878482c37b7656c5ae3ccc0f" Oct 30 19:00:01.998992 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:01.998817 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-03362d-29889780-hwd5x" podStartSLOduration=1.998795301 podStartE2EDuration="1.998795301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:01.99776749 +0000 UTC m=+1569.526798630" watchObservedRunningTime="2026-10-30 19:00:01.998795301 +0000 UTC m=+1569.527826439" Oct 30 19:00:02.848218 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:02.848101 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:02.947534 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:02.947496 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-86c150/auth-test-model-86c150-kserve-84bf95dfd5-5qc64" Oct 30 19:00:02.961886 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:02.961854 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-86c150/auth-test-model-86c150-kserve-84bf95dfd5-5qc64" Oct 30 19:00:02.992925 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:02.992888 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889780-qmbj9" event={"ID":"ef261723-21fe-493d-940f-394b9aca8fbc","Type":"ContainerStarted","Data":"85acba092a25db0f4cadac895d6bae8765cc6da2f22ee65f07dbe7c66d52d093"} Oct 30 19:00:02.994655 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:02.994626 6960 generic.go:356] "Generic (PLEG): container finished" podID="ba9e1a42-4e4d-4b4a-8a2b-28823c69e663" containerID="0f65aaf8b82d3f251b47d7037cae09f017d5b179b0dd86b4f5d65cf7b0c481c0" exitCode=60 Oct 30 19:00:02.994805 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:02.994710 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-03362d-29889780-hwd5x" event={"ID":"ba9e1a42-4e4d-4b4a-8a2b-28823c69e663","Type":"ContainerDied","Data":"0f65aaf8b82d3f251b47d7037cae09f017d5b179b0dd86b4f5d65cf7b0c481c0"} Oct 30 19:00:02.995224 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:02.995203 6960 scope.go:122] "RemoveContainer" containerID="0f65aaf8b82d3f251b47d7037cae09f017d5b179b0dd86b4f5d65cf7b0c481c0" Oct 30 19:00:02.996905 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:02.996879 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da70a5-29889780-dn54w" event={"ID":"734520a1-484d-4cc4-a340-302045e8ab82","Type":"ContainerStarted","Data":"b3615c9bd36725747012f4c906a81c754b008cb93a5b14994adc421c34ecba6e"} Oct 30 19:00:02.998636 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:02.998619 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ab7df0-29889780-z66fz" event={"ID":"0053dbc6-5623-498a-9fcf-9e89e1dc18ee","Type":"ContainerStarted","Data":"96c70904945f915d0fe0deba361e233cdac8024f89d210159bbd072a208f64cf"} Oct 30 19:00:03.011852 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:03.011832 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-86c150/auth-test-model-86c150-kserve-84bf95dfd5-5qc64" Oct 30 19:00:03.022064 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:03.022027 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889780-qmbj9" podStartSLOduration=3.022014637 podStartE2EDuration="3.022014637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:03.019499454 +0000 UTC m=+1570.548530592" watchObservedRunningTime="2026-10-30 19:00:03.022014637 +0000 UTC m=+1570.551045774" Oct 30 19:00:03.067748 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:03.067698 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da70a5-29889780-dn54w" podStartSLOduration=3.06768022 podStartE2EDuration="3.06768022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:03.057830277 +0000 UTC m=+1570.586861416" watchObservedRunningTime="2026-10-30 19:00:03.06768022 +0000 UTC m=+1570.596711360" Oct 30 19:00:03.082504 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:03.082461 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ab7df0-29889780-z66fz" podStartSLOduration=3.082447206 podStartE2EDuration="3.082447206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:03.080015404 +0000 UTC m=+1570.609046543" watchObservedRunningTime="2026-10-30 19:00:03.082447206 +0000 UTC m=+1570.611478343" Oct 30 19:00:03.841275 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:03.841237 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:04.008480 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:04.008446 6960 generic.go:356] "Generic (PLEG): container finished" podID="ef261723-21fe-493d-940f-394b9aca8fbc" containerID="85acba092a25db0f4cadac895d6bae8765cc6da2f22ee65f07dbe7c66d52d093" exitCode=60 Oct 30 19:00:04.008923 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:04.008521 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889780-qmbj9" event={"ID":"ef261723-21fe-493d-940f-394b9aca8fbc","Type":"ContainerDied","Data":"85acba092a25db0f4cadac895d6bae8765cc6da2f22ee65f07dbe7c66d52d093"} Oct 30 19:00:04.008923 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:04.008580 6960 scope.go:122] "RemoveContainer" containerID="06fdf63e4b70002d4a4e19427f9fd10342a9a57a878482c37b7656c5ae3ccc0f" Oct 30 19:00:04.008923 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:04.008883 6960 scope.go:122] "RemoveContainer" containerID="85acba092a25db0f4cadac895d6bae8765cc6da2f22ee65f07dbe7c66d52d093" Oct 30 19:00:04.009291 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:00:04.009266 6960 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889780-qmbj9_odh-ai-gateway-infra(ef261723-21fe-493d-940f-394b9aca8fbc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889780-qmbj9" podUID="ef261723-21fe-493d-940f-394b9aca8fbc" Oct 30 19:00:04.010866 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:04.010834 6960 generic.go:356] "Generic (PLEG): container finished" podID="ba9e1a42-4e4d-4b4a-8a2b-28823c69e663" containerID="81e39b39d10563adbb1ba35e712eb3530727c32d49ff092c761aec0b97a8af12" exitCode=60 Oct 30 19:00:04.010967 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:04.010918 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-03362d-29889780-hwd5x" event={"ID":"ba9e1a42-4e4d-4b4a-8a2b-28823c69e663","Type":"ContainerDied","Data":"81e39b39d10563adbb1ba35e712eb3530727c32d49ff092c761aec0b97a8af12"} Oct 30 19:00:04.011285 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:04.011267 6960 scope.go:122] "RemoveContainer" containerID="81e39b39d10563adbb1ba35e712eb3530727c32d49ff092c761aec0b97a8af12" Oct 30 19:00:04.011665 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:00:04.011641 6960 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-03362d-29889780-hwd5x_odh-ai-gateway-infra(ba9e1a42-4e4d-4b4a-8a2b-28823c69e663)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-03362d-29889780-hwd5x" podUID="ba9e1a42-4e4d-4b4a-8a2b-28823c69e663" Oct 30 19:00:04.012841 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:04.012822 6960 generic.go:356] "Generic (PLEG): container finished" podID="734520a1-484d-4cc4-a340-302045e8ab82" containerID="b3615c9bd36725747012f4c906a81c754b008cb93a5b14994adc421c34ecba6e" exitCode=60 Oct 30 19:00:04.012965 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:04.012852 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da70a5-29889780-dn54w" event={"ID":"734520a1-484d-4cc4-a340-302045e8ab82","Type":"ContainerDied","Data":"b3615c9bd36725747012f4c906a81c754b008cb93a5b14994adc421c34ecba6e"} Oct 30 19:00:04.013113 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:04.013093 6960 scope.go:122] "RemoveContainer" containerID="b3615c9bd36725747012f4c906a81c754b008cb93a5b14994adc421c34ecba6e" Oct 30 19:00:04.013418 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:00:04.013396 6960 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-da70a5-29889780-dn54w_odh-ai-gateway-infra(734520a1-484d-4cc4-a340-302045e8ab82)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da70a5-29889780-dn54w" podUID="734520a1-484d-4cc4-a340-302045e8ab82" Oct 30 19:00:04.014897 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:04.014879 6960 generic.go:356] "Generic (PLEG): container finished" podID="0053dbc6-5623-498a-9fcf-9e89e1dc18ee" containerID="96c70904945f915d0fe0deba361e233cdac8024f89d210159bbd072a208f64cf" exitCode=60 Oct 30 19:00:04.014972 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:04.014929 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ab7df0-29889780-z66fz" event={"ID":"0053dbc6-5623-498a-9fcf-9e89e1dc18ee","Type":"ContainerDied","Data":"96c70904945f915d0fe0deba361e233cdac8024f89d210159bbd072a208f64cf"} Oct 30 19:00:04.015195 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:04.015179 6960 scope.go:122] "RemoveContainer" containerID="96c70904945f915d0fe0deba361e233cdac8024f89d210159bbd072a208f64cf" Oct 30 19:00:04.015438 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:00:04.015421 6960 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-ab7df0-29889780-z66fz_odh-ai-gateway-infra(0053dbc6-5623-498a-9fcf-9e89e1dc18ee)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ab7df0-29889780-z66fz" podUID="0053dbc6-5623-498a-9fcf-9e89e1dc18ee" Oct 30 19:00:04.032931 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:04.032898 6960 scope.go:122] "RemoveContainer" containerID="0f65aaf8b82d3f251b47d7037cae09f017d5b179b0dd86b4f5d65cf7b0c481c0" Oct 30 19:00:04.046968 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:04.046882 6960 scope.go:122] "RemoveContainer" containerID="0fd068d699d8997ef4d671a9a56ca423a17d260ee4d4192231a1e8e525fd7ba5" Oct 30 19:00:04.060959 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:04.060909 6960 scope.go:122] "RemoveContainer" containerID="a203e30b15b62b40471f137b9afd553f91cb9324b23ad8af35846823d7e1ae25" Oct 30 19:00:04.825018 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:04.824975 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:05.022686 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:05.022644 6960 scope.go:122] "RemoveContainer" containerID="85acba092a25db0f4cadac895d6bae8765cc6da2f22ee65f07dbe7c66d52d093" Oct 30 19:00:05.023119 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:00:05.023035 6960 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889780-qmbj9_odh-ai-gateway-infra(ef261723-21fe-493d-940f-394b9aca8fbc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889780-qmbj9" podUID="ef261723-21fe-493d-940f-394b9aca8fbc" Oct 30 19:00:05.024426 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:05.024404 6960 scope.go:122] "RemoveContainer" containerID="81e39b39d10563adbb1ba35e712eb3530727c32d49ff092c761aec0b97a8af12" Oct 30 19:00:05.024691 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:00:05.024673 6960 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-03362d-29889780-hwd5x_odh-ai-gateway-infra(ba9e1a42-4e4d-4b4a-8a2b-28823c69e663)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-03362d-29889780-hwd5x" podUID="ba9e1a42-4e4d-4b4a-8a2b-28823c69e663" Oct 30 19:00:05.026065 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:05.026041 6960 scope.go:122] "RemoveContainer" containerID="b3615c9bd36725747012f4c906a81c754b008cb93a5b14994adc421c34ecba6e" Oct 30 19:00:05.026404 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:00:05.026373 6960 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-da70a5-29889780-dn54w_odh-ai-gateway-infra(734520a1-484d-4cc4-a340-302045e8ab82)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da70a5-29889780-dn54w" podUID="734520a1-484d-4cc4-a340-302045e8ab82" Oct 30 19:00:05.027688 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:05.027672 6960 scope.go:122] "RemoveContainer" containerID="96c70904945f915d0fe0deba361e233cdac8024f89d210159bbd072a208f64cf" Oct 30 19:00:05.027887 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:00:05.027871 6960 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-ab7df0-29889780-z66fz_odh-ai-gateway-infra(0053dbc6-5623-498a-9fcf-9e89e1dc18ee)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ab7df0-29889780-z66fz" podUID="0053dbc6-5623-498a-9fcf-9e89e1dc18ee" Oct 30 19:00:05.776257 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:05.776226 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:06.667170 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:06.664235 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-743e0b-646fdd7b4f-5znrc"] Oct 30 19:00:06.681182 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:06.675391 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-743e0b-646fdd7b4f-5znrc"] Oct 30 19:00:06.681182 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:06.675512 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-743e0b-646fdd7b4f-5znrc" Oct 30 19:00:06.684566 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:06.684544 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-743e0b-serving-cert\"" Oct 30 19:00:06.784864 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:06.784831 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6218b805-ff40-4951-a4e2-5df7cd1e4102-maas-api-tls\") pod \"maas-api-e2e-explicit-743e0b-646fdd7b4f-5znrc\" (UID: \"6218b805-ff40-4951-a4e2-5df7cd1e4102\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-743e0b-646fdd7b4f-5znrc" Oct 30 19:00:06.785021 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:06.784999 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6218b805-ff40-4951-a4e2-5df7cd1e4102-metrics-certs\") pod \"maas-api-e2e-explicit-743e0b-646fdd7b4f-5znrc\" (UID: \"6218b805-ff40-4951-a4e2-5df7cd1e4102\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-743e0b-646fdd7b4f-5znrc" Oct 30 19:00:06.785071 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:06.785052 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6218b805-ff40-4951-a4e2-5df7cd1e4102-openshift-service-ca\") pod \"maas-api-e2e-explicit-743e0b-646fdd7b4f-5znrc\" (UID: \"6218b805-ff40-4951-a4e2-5df7cd1e4102\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-743e0b-646fdd7b4f-5znrc" Oct 30 19:00:06.785161 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:06.785113 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4k8p\" (UniqueName: \"kubernetes.io/projected/6218b805-ff40-4951-a4e2-5df7cd1e4102-kube-api-access-v4k8p\") pod \"maas-api-e2e-explicit-743e0b-646fdd7b4f-5znrc\" (UID: \"6218b805-ff40-4951-a4e2-5df7cd1e4102\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-743e0b-646fdd7b4f-5znrc" Oct 30 19:00:06.795632 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:06.795602 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:06.887008 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:06.886941 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6218b805-ff40-4951-a4e2-5df7cd1e4102-metrics-certs\") pod \"maas-api-e2e-explicit-743e0b-646fdd7b4f-5znrc\" (UID: \"6218b805-ff40-4951-a4e2-5df7cd1e4102\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-743e0b-646fdd7b4f-5znrc" Oct 30 19:00:06.887343 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:06.887318 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6218b805-ff40-4951-a4e2-5df7cd1e4102-openshift-service-ca\") pod \"maas-api-e2e-explicit-743e0b-646fdd7b4f-5znrc\" (UID: \"6218b805-ff40-4951-a4e2-5df7cd1e4102\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-743e0b-646fdd7b4f-5znrc" Oct 30 19:00:06.887523 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:06.887506 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4k8p\" (UniqueName: \"kubernetes.io/projected/6218b805-ff40-4951-a4e2-5df7cd1e4102-kube-api-access-v4k8p\") pod \"maas-api-e2e-explicit-743e0b-646fdd7b4f-5znrc\" (UID: \"6218b805-ff40-4951-a4e2-5df7cd1e4102\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-743e0b-646fdd7b4f-5znrc" Oct 30 19:00:06.887823 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:06.887802 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6218b805-ff40-4951-a4e2-5df7cd1e4102-maas-api-tls\") pod \"maas-api-e2e-explicit-743e0b-646fdd7b4f-5znrc\" (UID: \"6218b805-ff40-4951-a4e2-5df7cd1e4102\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-743e0b-646fdd7b4f-5znrc" Oct 30 19:00:06.888103 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:06.888078 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6218b805-ff40-4951-a4e2-5df7cd1e4102-openshift-service-ca\") pod \"maas-api-e2e-explicit-743e0b-646fdd7b4f-5znrc\" (UID: \"6218b805-ff40-4951-a4e2-5df7cd1e4102\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-743e0b-646fdd7b4f-5znrc" Oct 30 19:00:06.888929 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:00:06.888905 6960 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-743e0b-serving-cert: secret "maas-api-e2e-explicit-743e0b-serving-cert" not found Oct 30 19:00:06.889116 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:00:06.889103 6960 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6218b805-ff40-4951-a4e2-5df7cd1e4102-maas-api-tls podName:6218b805-ff40-4951-a4e2-5df7cd1e4102 nodeName:}" failed. No retries permitted until 2026-10-30 19:00:07.389079598 +0000 UTC m=+1574.918110735 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6218b805-ff40-4951-a4e2-5df7cd1e4102-maas-api-tls") pod "maas-api-e2e-explicit-743e0b-646fdd7b4f-5znrc" (UID: "6218b805-ff40-4951-a4e2-5df7cd1e4102") : secret "maas-api-e2e-explicit-743e0b-serving-cert" not found Oct 30 19:00:06.892165 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:06.892118 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6218b805-ff40-4951-a4e2-5df7cd1e4102-metrics-certs\") pod \"maas-api-e2e-explicit-743e0b-646fdd7b4f-5znrc\" (UID: \"6218b805-ff40-4951-a4e2-5df7cd1e4102\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-743e0b-646fdd7b4f-5znrc" Oct 30 19:00:06.903136 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:06.903110 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4k8p\" (UniqueName: \"kubernetes.io/projected/6218b805-ff40-4951-a4e2-5df7cd1e4102-kube-api-access-v4k8p\") pod \"maas-api-e2e-explicit-743e0b-646fdd7b4f-5znrc\" (UID: \"6218b805-ff40-4951-a4e2-5df7cd1e4102\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-743e0b-646fdd7b4f-5znrc" Oct 30 19:00:06.944479 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:06.944091 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:07.096579 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:07.096549 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:07.397171 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:07.396492 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6218b805-ff40-4951-a4e2-5df7cd1e4102-maas-api-tls\") pod \"maas-api-e2e-explicit-743e0b-646fdd7b4f-5znrc\" (UID: \"6218b805-ff40-4951-a4e2-5df7cd1e4102\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-743e0b-646fdd7b4f-5znrc" Oct 30 19:00:07.405201 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:07.405127 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6218b805-ff40-4951-a4e2-5df7cd1e4102-maas-api-tls\") pod \"maas-api-e2e-explicit-743e0b-646fdd7b4f-5znrc\" (UID: \"6218b805-ff40-4951-a4e2-5df7cd1e4102\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-743e0b-646fdd7b4f-5znrc" Oct 30 19:00:07.603649 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:07.602400 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-743e0b-646fdd7b4f-5znrc" Oct 30 19:00:07.933220 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:07.933193 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-743e0b-646fdd7b4f-5znrc"] Oct 30 19:00:07.934602 ip-10-0-140-163 kubenswrapper[6960]: W1030 19:00:07.934578 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6218b805_ff40_4951_a4e2_5df7cd1e4102.slice/crio-8c2313551386a1b5b57eb31429e85ae77167f1ea6672cf05285a77afcab9c93d WatchSource:0}: Error finding container 8c2313551386a1b5b57eb31429e85ae77167f1ea6672cf05285a77afcab9c93d: Status 404 returned error can't find the container with id 8c2313551386a1b5b57eb31429e85ae77167f1ea6672cf05285a77afcab9c93d Oct 30 19:00:07.948397 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:07.948377 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-743e0b-85c6498fc5-hdpjx" Oct 30 19:00:08.054033 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:08.053955 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-743e0b-646fdd7b4f-5znrc" event={"ID":"6218b805-ff40-4951-a4e2-5df7cd1e4102","Type":"ContainerStarted","Data":"8c2313551386a1b5b57eb31429e85ae77167f1ea6672cf05285a77afcab9c93d"} Oct 30 19:00:08.111662 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:08.111574 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:08.251333 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:08.251284 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:09.063219 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:09.063181 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-743e0b-646fdd7b4f-5znrc" event={"ID":"6218b805-ff40-4951-a4e2-5df7cd1e4102","Type":"ContainerStarted","Data":"baacb588eeede2a272f08a2b5fd946377302f5cdedd63b923aa3ed2c2fdaa6d6"} Oct 30 19:00:09.063667 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:09.063379 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-743e0b-646fdd7b4f-5znrc" Oct 30 19:00:09.085931 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:09.085885 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-743e0b-646fdd7b4f-5znrc" podStartSLOduration=2.52598401 podStartE2EDuration="3.085867666s" totalImagesPullingTime="559.883656ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:09.084454634 +0000 UTC m=+1576.613485772" watchObservedRunningTime="2026-10-30 19:00:09.085867666 +0000 UTC m=+1576.614898805" Oct 30 19:00:09.146782 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:09.146746 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:09.517457 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:09.517379 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:09.947778 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:09.947750 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-784fbf-6cdcf7dbc8-tq592"] Oct 30 19:00:09.952686 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:09.952659 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-784fbf-6cdcf7dbc8-tq592" Oct 30 19:00:09.967733 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:09.966210 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-784fbf-serving-cert\"" Oct 30 19:00:09.978168 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:09.976787 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-784fbf-6cdcf7dbc8-tq592"] Oct 30 19:00:10.033810 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:10.033775 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:10.047079 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:10.047048 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c610dae0-c986-4c71-b430-9e1e7bd73e9b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-784fbf-6cdcf7dbc8-tq592\" (UID: \"c610dae0-c986-4c71-b430-9e1e7bd73e9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-784fbf-6cdcf7dbc8-tq592" Oct 30 19:00:10.047320 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:10.047109 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c610dae0-c986-4c71-b430-9e1e7bd73e9b-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-784fbf-6cdcf7dbc8-tq592\" (UID: \"c610dae0-c986-4c71-b430-9e1e7bd73e9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-784fbf-6cdcf7dbc8-tq592" Oct 30 19:00:10.047402 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:10.047362 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8bg4\" (UniqueName: \"kubernetes.io/projected/c610dae0-c986-4c71-b430-9e1e7bd73e9b-kube-api-access-f8bg4\") pod \"maas-api-e2e-shared-b-w1-784fbf-6cdcf7dbc8-tq592\" (UID: \"c610dae0-c986-4c71-b430-9e1e7bd73e9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-784fbf-6cdcf7dbc8-tq592" Oct 30 19:00:10.047444 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:10.047414 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c610dae0-c986-4c71-b430-9e1e7bd73e9b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-784fbf-6cdcf7dbc8-tq592\" (UID: \"c610dae0-c986-4c71-b430-9e1e7bd73e9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-784fbf-6cdcf7dbc8-tq592" Oct 30 19:00:10.152219 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:10.150523 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8bg4\" (UniqueName: \"kubernetes.io/projected/c610dae0-c986-4c71-b430-9e1e7bd73e9b-kube-api-access-f8bg4\") pod \"maas-api-e2e-shared-b-w1-784fbf-6cdcf7dbc8-tq592\" (UID: \"c610dae0-c986-4c71-b430-9e1e7bd73e9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-784fbf-6cdcf7dbc8-tq592" Oct 30 19:00:10.152219 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:10.150609 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c610dae0-c986-4c71-b430-9e1e7bd73e9b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-784fbf-6cdcf7dbc8-tq592\" (UID: \"c610dae0-c986-4c71-b430-9e1e7bd73e9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-784fbf-6cdcf7dbc8-tq592" Oct 30 19:00:10.152219 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:10.150902 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c610dae0-c986-4c71-b430-9e1e7bd73e9b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-784fbf-6cdcf7dbc8-tq592\" (UID: \"c610dae0-c986-4c71-b430-9e1e7bd73e9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-784fbf-6cdcf7dbc8-tq592" Oct 30 19:00:10.152219 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:10.150954 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c610dae0-c986-4c71-b430-9e1e7bd73e9b-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-784fbf-6cdcf7dbc8-tq592\" (UID: \"c610dae0-c986-4c71-b430-9e1e7bd73e9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-784fbf-6cdcf7dbc8-tq592" Oct 30 19:00:10.161411 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:10.160709 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c610dae0-c986-4c71-b430-9e1e7bd73e9b-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-784fbf-6cdcf7dbc8-tq592\" (UID: \"c610dae0-c986-4c71-b430-9e1e7bd73e9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-784fbf-6cdcf7dbc8-tq592" Oct 30 19:00:10.161411 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:00:10.161367 6960 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-784fbf-serving-cert: secret "maas-api-e2e-shared-b-w1-784fbf-serving-cert" not found Oct 30 19:00:10.161411 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:00:10.161432 6960 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c610dae0-c986-4c71-b430-9e1e7bd73e9b-maas-api-tls podName:c610dae0-c986-4c71-b430-9e1e7bd73e9b nodeName:}" failed. No retries permitted until 2026-10-30 19:00:10.661412399 +0000 UTC m=+1578.190443521 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c610dae0-c986-4c71-b430-9e1e7bd73e9b-maas-api-tls") pod "maas-api-e2e-shared-b-w1-784fbf-6cdcf7dbc8-tq592" (UID: "c610dae0-c986-4c71-b430-9e1e7bd73e9b") : secret "maas-api-e2e-shared-b-w1-784fbf-serving-cert" not found Oct 30 19:00:10.162731 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:10.162707 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c610dae0-c986-4c71-b430-9e1e7bd73e9b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-784fbf-6cdcf7dbc8-tq592\" (UID: \"c610dae0-c986-4c71-b430-9e1e7bd73e9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-784fbf-6cdcf7dbc8-tq592" Oct 30 19:00:10.176522 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:10.176497 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8bg4\" (UniqueName: \"kubernetes.io/projected/c610dae0-c986-4c71-b430-9e1e7bd73e9b-kube-api-access-f8bg4\") pod \"maas-api-e2e-shared-b-w1-784fbf-6cdcf7dbc8-tq592\" (UID: \"c610dae0-c986-4c71-b430-9e1e7bd73e9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-784fbf-6cdcf7dbc8-tq592" Oct 30 19:00:10.188300 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:10.188265 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:10.346549 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:10.346513 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:10.763846 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:10.763742 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c610dae0-c986-4c71-b430-9e1e7bd73e9b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-784fbf-6cdcf7dbc8-tq592\" (UID: \"c610dae0-c986-4c71-b430-9e1e7bd73e9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-784fbf-6cdcf7dbc8-tq592" Oct 30 19:00:10.780759 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:10.780729 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c610dae0-c986-4c71-b430-9e1e7bd73e9b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-784fbf-6cdcf7dbc8-tq592\" (UID: \"c610dae0-c986-4c71-b430-9e1e7bd73e9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-784fbf-6cdcf7dbc8-tq592" Oct 30 19:00:10.877603 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:10.877572 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-784fbf-6cdcf7dbc8-tq592" Oct 30 19:00:11.201497 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:11.201468 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-784fbf-6cdcf7dbc8-tq592"] Oct 30 19:00:11.203482 ip-10-0-140-163 kubenswrapper[6960]: W1030 19:00:11.203451 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc610dae0_c986_4c71_b430_9e1e7bd73e9b.slice/crio-39770f40e993e93d9c4726d815570c6f01efb8c0a8f4c5dc1428d23ec8fa563f WatchSource:0}: Error finding container 39770f40e993e93d9c4726d815570c6f01efb8c0a8f4c5dc1428d23ec8fa563f: Status 404 returned error can't find the container with id 39770f40e993e93d9c4726d815570c6f01efb8c0a8f4c5dc1428d23ec8fa563f Oct 30 19:00:11.510281 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:11.510189 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:12.090193 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:12.090126 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-784fbf-6cdcf7dbc8-tq592" event={"ID":"c610dae0-c986-4c71-b430-9e1e7bd73e9b","Type":"ContainerStarted","Data":"a15ee0974b3f7189c18e196e6d97b15c4dec3cc481a3da02ccd10e173a5ad673"} Oct 30 19:00:12.090376 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:12.090219 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-784fbf-6cdcf7dbc8-tq592" Oct 30 19:00:12.090376 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:12.090236 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-784fbf-6cdcf7dbc8-tq592" event={"ID":"c610dae0-c986-4c71-b430-9e1e7bd73e9b","Type":"ContainerStarted","Data":"39770f40e993e93d9c4726d815570c6f01efb8c0a8f4c5dc1428d23ec8fa563f"} Oct 30 19:00:12.120634 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:12.120585 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-784fbf-6cdcf7dbc8-tq592" podStartSLOduration=2.711478605 podStartE2EDuration="3.120572536s" totalImagesPullingTime="409.093931ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:12.116951322 +0000 UTC m=+1579.645982472" watchObservedRunningTime="2026-10-30 19:00:12.120572536 +0000 UTC m=+1579.649603673" Oct 30 19:00:12.635197 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:12.635156 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:12.771489 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:12.771455 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:13.541970 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:13.541935 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:15.077239 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:15.077211 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-743e0b-646fdd7b4f-5znrc" Oct 30 19:00:15.585402 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:15.585373 6960 scope.go:122] "RemoveContainer" containerID="81e39b39d10563adbb1ba35e712eb3530727c32d49ff092c761aec0b97a8af12" Oct 30 19:00:15.585564 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:15.585510 6960 scope.go:122] "RemoveContainer" containerID="b3615c9bd36725747012f4c906a81c754b008cb93a5b14994adc421c34ecba6e" Oct 30 19:00:15.943818 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:15.943778 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:16.079004 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:16.078969 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:16.121011 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:16.120942 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-03362d-29889780-hwd5x" event={"ID":"ba9e1a42-4e4d-4b4a-8a2b-28823c69e663","Type":"ContainerStarted","Data":"7d1beaecc58262c10abf423315ea013574e561315ac63c9b0965a2ce50a1f90f"} Oct 30 19:00:16.123048 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:16.123026 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da70a5-29889780-dn54w" event={"ID":"734520a1-484d-4cc4-a340-302045e8ab82","Type":"ContainerStarted","Data":"c22f65a4c6cdbda17f1a9beee0a496178bd7f073b89b1c8ad7c3ba25f5ce2399"} Oct 30 19:00:16.229577 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:16.229536 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:16.586002 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:16.585971 6960 scope.go:122] "RemoveContainer" containerID="85acba092a25db0f4cadac895d6bae8765cc6da2f22ee65f07dbe7c66d52d093" Oct 30 19:00:16.586179 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:16.586121 6960 scope.go:122] "RemoveContainer" containerID="96c70904945f915d0fe0deba361e233cdac8024f89d210159bbd072a208f64cf" Oct 30 19:00:17.135459 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:17.135369 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ab7df0-29889780-z66fz" event={"ID":"0053dbc6-5623-498a-9fcf-9e89e1dc18ee","Type":"ContainerStarted","Data":"053ea6a9018cb1d1033f2d0fa17d157338345e9c6273c42b9ecd73efa42f055a"} Oct 30 19:00:17.137819 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:17.137786 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889780-qmbj9" event={"ID":"ef261723-21fe-493d-940f-394b9aca8fbc","Type":"ContainerStarted","Data":"8c3d75430323c7949dd1c24f31682738b9d380e72509285b64ac2f9b1c0980c4"} Oct 30 19:00:17.140715 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:17.140685 6960 generic.go:356] "Generic (PLEG): container finished" podID="ba9e1a42-4e4d-4b4a-8a2b-28823c69e663" containerID="7d1beaecc58262c10abf423315ea013574e561315ac63c9b0965a2ce50a1f90f" exitCode=60 Oct 30 19:00:17.140871 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:17.140753 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-03362d-29889780-hwd5x" event={"ID":"ba9e1a42-4e4d-4b4a-8a2b-28823c69e663","Type":"ContainerDied","Data":"7d1beaecc58262c10abf423315ea013574e561315ac63c9b0965a2ce50a1f90f"} Oct 30 19:00:17.140871 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:17.140809 6960 scope.go:122] "RemoveContainer" containerID="81e39b39d10563adbb1ba35e712eb3530727c32d49ff092c761aec0b97a8af12" Oct 30 19:00:17.141073 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:17.141062 6960 scope.go:122] "RemoveContainer" containerID="7d1beaecc58262c10abf423315ea013574e561315ac63c9b0965a2ce50a1f90f" Oct 30 19:00:17.141467 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:00:17.141441 6960 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-03362d-29889780-hwd5x_odh-ai-gateway-infra(ba9e1a42-4e4d-4b4a-8a2b-28823c69e663)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-03362d-29889780-hwd5x" podUID="ba9e1a42-4e4d-4b4a-8a2b-28823c69e663" Oct 30 19:00:17.143089 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:17.143065 6960 generic.go:356] "Generic (PLEG): container finished" podID="734520a1-484d-4cc4-a340-302045e8ab82" containerID="c22f65a4c6cdbda17f1a9beee0a496178bd7f073b89b1c8ad7c3ba25f5ce2399" exitCode=60 Oct 30 19:00:17.143222 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:17.143105 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da70a5-29889780-dn54w" event={"ID":"734520a1-484d-4cc4-a340-302045e8ab82","Type":"ContainerDied","Data":"c22f65a4c6cdbda17f1a9beee0a496178bd7f073b89b1c8ad7c3ba25f5ce2399"} Oct 30 19:00:17.143480 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:17.143466 6960 scope.go:122] "RemoveContainer" containerID="c22f65a4c6cdbda17f1a9beee0a496178bd7f073b89b1c8ad7c3ba25f5ce2399" Oct 30 19:00:17.143739 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:00:17.143715 6960 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-da70a5-29889780-dn54w_odh-ai-gateway-infra(734520a1-484d-4cc4-a340-302045e8ab82)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da70a5-29889780-dn54w" podUID="734520a1-484d-4cc4-a340-302045e8ab82" Oct 30 19:00:17.155309 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:17.155282 6960 scope.go:122] "RemoveContainer" containerID="b3615c9bd36725747012f4c906a81c754b008cb93a5b14994adc421c34ecba6e" Oct 30 19:00:17.164280 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:17.164243 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-03362d-29889780-hwd5x"] Oct 30 19:00:17.196408 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:17.196379 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da70a5-29889780-dn54w"] Oct 30 19:00:17.237616 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:17.237557 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:18.103312 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:18.103271 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-784fbf-6cdcf7dbc8-tq592" Oct 30 19:00:18.161368 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:18.161308 6960 generic.go:356] "Generic (PLEG): container finished" podID="0053dbc6-5623-498a-9fcf-9e89e1dc18ee" containerID="053ea6a9018cb1d1033f2d0fa17d157338345e9c6273c42b9ecd73efa42f055a" exitCode=60 Oct 30 19:00:18.161766 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:18.161497 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ab7df0-29889780-z66fz" event={"ID":"0053dbc6-5623-498a-9fcf-9e89e1dc18ee","Type":"ContainerDied","Data":"053ea6a9018cb1d1033f2d0fa17d157338345e9c6273c42b9ecd73efa42f055a"} Oct 30 19:00:18.161766 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:18.161579 6960 scope.go:122] "RemoveContainer" containerID="96c70904945f915d0fe0deba361e233cdac8024f89d210159bbd072a208f64cf" Oct 30 19:00:18.162083 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:18.162061 6960 scope.go:122] "RemoveContainer" containerID="053ea6a9018cb1d1033f2d0fa17d157338345e9c6273c42b9ecd73efa42f055a" Oct 30 19:00:18.164809 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:00:18.162480 6960 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-ab7df0-29889780-z66fz_odh-ai-gateway-infra(0053dbc6-5623-498a-9fcf-9e89e1dc18ee)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ab7df0-29889780-z66fz" podUID="0053dbc6-5623-498a-9fcf-9e89e1dc18ee" Oct 30 19:00:18.166637 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:18.166537 6960 generic.go:356] "Generic (PLEG): container finished" podID="ef261723-21fe-493d-940f-394b9aca8fbc" containerID="8c3d75430323c7949dd1c24f31682738b9d380e72509285b64ac2f9b1c0980c4" exitCode=60 Oct 30 19:00:18.166803 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:18.166648 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889780-qmbj9" event={"ID":"ef261723-21fe-493d-940f-394b9aca8fbc","Type":"ContainerDied","Data":"8c3d75430323c7949dd1c24f31682738b9d380e72509285b64ac2f9b1c0980c4"} Oct 30 19:00:18.167234 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:18.167193 6960 scope.go:122] "RemoveContainer" containerID="8c3d75430323c7949dd1c24f31682738b9d380e72509285b64ac2f9b1c0980c4" Oct 30 19:00:18.167622 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:00:18.167591 6960 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889780-qmbj9_odh-ai-gateway-infra(ef261723-21fe-493d-940f-394b9aca8fbc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889780-qmbj9" podUID="ef261723-21fe-493d-940f-394b9aca8fbc" Oct 30 19:00:18.189846 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:18.189823 6960 scope.go:122] "RemoveContainer" containerID="85acba092a25db0f4cadac895d6bae8765cc6da2f22ee65f07dbe7c66d52d093" Oct 30 19:00:18.195017 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:18.194964 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ab7df0-29889780-z66fz"] Oct 30 19:00:18.208268 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:18.208229 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:18.231286 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:18.231256 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889780-qmbj9"] Oct 30 19:00:18.419851 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:18.419827 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-03362d-29889780-hwd5x" Oct 30 19:00:18.450224 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:18.450201 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da70a5-29889780-dn54w" Oct 30 19:00:18.497678 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:18.497650 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba9e1a42-4e4d-4b4a-8a2b-28823c69e663-kube-api-access-vvh5d\" (UniqueName: \"kubernetes.io/projected/ba9e1a42-4e4d-4b4a-8a2b-28823c69e663-kube-api-access-vvh5d\") pod \"ba9e1a42-4e4d-4b4a-8a2b-28823c69e663\" (UID: \"ba9e1a42-4e4d-4b4a-8a2b-28823c69e663\") " Oct 30 19:00:18.497853 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:18.497696 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/734520a1-484d-4cc4-a340-302045e8ab82-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/734520a1-484d-4cc4-a340-302045e8ab82-maas-api-service-ca\") pod \"734520a1-484d-4cc4-a340-302045e8ab82\" (UID: \"734520a1-484d-4cc4-a340-302045e8ab82\") " Oct 30 19:00:18.497853 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:18.497784 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba9e1a42-4e4d-4b4a-8a2b-28823c69e663-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba9e1a42-4e4d-4b4a-8a2b-28823c69e663-maas-api-service-ca\") pod \"ba9e1a42-4e4d-4b4a-8a2b-28823c69e663\" (UID: \"ba9e1a42-4e4d-4b4a-8a2b-28823c69e663\") " Oct 30 19:00:18.498052 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:18.498028 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/734520a1-484d-4cc4-a340-302045e8ab82-kube-api-access-w4snc\" (UniqueName: \"kubernetes.io/projected/734520a1-484d-4cc4-a340-302045e8ab82-kube-api-access-w4snc\") pod \"734520a1-484d-4cc4-a340-302045e8ab82\" (UID: \"734520a1-484d-4cc4-a340-302045e8ab82\") " Oct 30 19:00:18.498195 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:18.498169 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/734520a1-484d-4cc4-a340-302045e8ab82-maas-api-service-ca" pod "734520a1-484d-4cc4-a340-302045e8ab82" (UID: "734520a1-484d-4cc4-a340-302045e8ab82"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:00:18.498268 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:18.498200 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba9e1a42-4e4d-4b4a-8a2b-28823c69e663-maas-api-service-ca" pod "ba9e1a42-4e4d-4b4a-8a2b-28823c69e663" (UID: "ba9e1a42-4e4d-4b4a-8a2b-28823c69e663"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:00:18.498869 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:18.498848 6960 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/734520a1-484d-4cc4-a340-302045e8ab82-maas-api-service-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:00:18.498973 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:18.498874 6960 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba9e1a42-4e4d-4b4a-8a2b-28823c69e663-maas-api-service-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:00:18.500574 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:18.500548 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba9e1a42-4e4d-4b4a-8a2b-28823c69e663-kube-api-access-vvh5d" pod "ba9e1a42-4e4d-4b4a-8a2b-28823c69e663" (UID: "ba9e1a42-4e4d-4b4a-8a2b-28823c69e663"). InnerVolumeSpecName "kube-api-access-vvh5d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:18.500804 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:18.500783 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/734520a1-484d-4cc4-a340-302045e8ab82-kube-api-access-w4snc" pod "734520a1-484d-4cc4-a340-302045e8ab82" (UID: "734520a1-484d-4cc4-a340-302045e8ab82"). InnerVolumeSpecName "kube-api-access-w4snc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:18.600611 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:18.600581 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4snc\" (UniqueName: \"kubernetes.io/projected/734520a1-484d-4cc4-a340-302045e8ab82-kube-api-access-w4snc\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:00:18.600611 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:18.600612 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvh5d\" (UniqueName: \"kubernetes.io/projected/ba9e1a42-4e4d-4b4a-8a2b-28823c69e663-kube-api-access-vvh5d\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:00:19.176246 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:19.176202 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-03362d-29889780-hwd5x" event={"ID":"ba9e1a42-4e4d-4b4a-8a2b-28823c69e663","Type":"ContainerDied","Data":"3a3ac0947f0ff4d83d22110adcee7a9051377b906fe9d94f564fadcaf6e3917a"} Oct 30 19:00:19.176746 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:19.176300 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-03362d-29889780-hwd5x" Oct 30 19:00:19.176848 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:19.176310 6960 scope.go:122] "RemoveContainer" containerID="7d1beaecc58262c10abf423315ea013574e561315ac63c9b0965a2ce50a1f90f" Oct 30 19:00:19.178926 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:19.178899 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da70a5-29889780-dn54w" event={"ID":"734520a1-484d-4cc4-a340-302045e8ab82","Type":"ContainerDied","Data":"106de254aa0a48248411234e80567380b2bf5cc4e883fb5512fb5535846e34f0"} Oct 30 19:00:19.179202 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:19.179183 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da70a5-29889780-dn54w" Oct 30 19:00:19.193337 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:19.193314 6960 scope.go:122] "RemoveContainer" containerID="c22f65a4c6cdbda17f1a9beee0a496178bd7f073b89b1c8ad7c3ba25f5ce2399" Oct 30 19:00:19.230191 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:19.230109 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:19.269736 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:19.269205 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-03362d-29889780-hwd5x"] Oct 30 19:00:19.283543 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:19.283511 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-03362d-29889780-hwd5x"] Oct 30 19:00:19.314837 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:19.314800 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da70a5-29889780-dn54w"] Oct 30 19:00:19.319383 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:19.319358 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-da70a5-29889780-dn54w"] Oct 30 19:00:19.422510 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:19.422490 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ab7df0-29889780-z66fz" Oct 30 19:00:19.424751 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:19.424732 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889780-qmbj9" Oct 30 19:00:19.514263 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:19.514191 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef261723-21fe-493d-940f-394b9aca8fbc-kube-api-access-vs6n4\" (UniqueName: \"kubernetes.io/projected/ef261723-21fe-493d-940f-394b9aca8fbc-kube-api-access-vs6n4\") pod \"ef261723-21fe-493d-940f-394b9aca8fbc\" (UID: \"ef261723-21fe-493d-940f-394b9aca8fbc\") " Oct 30 19:00:19.514396 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:19.514309 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0053dbc6-5623-498a-9fcf-9e89e1dc18ee-kube-api-access-bfgpb\" (UniqueName: \"kubernetes.io/projected/0053dbc6-5623-498a-9fcf-9e89e1dc18ee-kube-api-access-bfgpb\") pod \"0053dbc6-5623-498a-9fcf-9e89e1dc18ee\" (UID: \"0053dbc6-5623-498a-9fcf-9e89e1dc18ee\") " Oct 30 19:00:19.514447 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:19.514404 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef261723-21fe-493d-940f-394b9aca8fbc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef261723-21fe-493d-940f-394b9aca8fbc-maas-api-service-ca\") pod \"ef261723-21fe-493d-940f-394b9aca8fbc\" (UID: \"ef261723-21fe-493d-940f-394b9aca8fbc\") " Oct 30 19:00:19.514447 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:19.514431 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0053dbc6-5623-498a-9fcf-9e89e1dc18ee-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0053dbc6-5623-498a-9fcf-9e89e1dc18ee-maas-api-service-ca\") pod \"0053dbc6-5623-498a-9fcf-9e89e1dc18ee\" (UID: \"0053dbc6-5623-498a-9fcf-9e89e1dc18ee\") " Oct 30 19:00:19.514752 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:19.514728 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef261723-21fe-493d-940f-394b9aca8fbc-maas-api-service-ca" pod "ef261723-21fe-493d-940f-394b9aca8fbc" (UID: "ef261723-21fe-493d-940f-394b9aca8fbc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:00:19.514871 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:19.514795 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0053dbc6-5623-498a-9fcf-9e89e1dc18ee-maas-api-service-ca" pod "0053dbc6-5623-498a-9fcf-9e89e1dc18ee" (UID: "0053dbc6-5623-498a-9fcf-9e89e1dc18ee"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:00:19.514994 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:19.514975 6960 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef261723-21fe-493d-940f-394b9aca8fbc-maas-api-service-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:00:19.515044 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:19.515000 6960 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0053dbc6-5623-498a-9fcf-9e89e1dc18ee-maas-api-service-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:00:19.516510 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:19.516493 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef261723-21fe-493d-940f-394b9aca8fbc-kube-api-access-vs6n4" pod "ef261723-21fe-493d-940f-394b9aca8fbc" (UID: "ef261723-21fe-493d-940f-394b9aca8fbc"). InnerVolumeSpecName "kube-api-access-vs6n4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:19.516614 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:19.516596 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0053dbc6-5623-498a-9fcf-9e89e1dc18ee-kube-api-access-bfgpb" pod "0053dbc6-5623-498a-9fcf-9e89e1dc18ee" (UID: "0053dbc6-5623-498a-9fcf-9e89e1dc18ee"). InnerVolumeSpecName "kube-api-access-bfgpb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:19.616158 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:19.616126 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfgpb\" (UniqueName: \"kubernetes.io/projected/0053dbc6-5623-498a-9fcf-9e89e1dc18ee-kube-api-access-bfgpb\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:00:19.616316 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:19.616165 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vs6n4\" (UniqueName: \"kubernetes.io/projected/ef261723-21fe-493d-940f-394b9aca8fbc-kube-api-access-vs6n4\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:00:20.197022 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:20.196994 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ab7df0-29889780-z66fz" Oct 30 19:00:20.197501 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:20.197029 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ab7df0-29889780-z66fz" event={"ID":"0053dbc6-5623-498a-9fcf-9e89e1dc18ee","Type":"ContainerDied","Data":"f4ef1c57ccabd9975a6e823515ca4c86b99886f27fea8f9e5e5357148bf8a4a0"} Oct 30 19:00:20.197501 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:20.197090 6960 scope.go:122] "RemoveContainer" containerID="053ea6a9018cb1d1033f2d0fa17d157338345e9c6273c42b9ecd73efa42f055a" Oct 30 19:00:20.198604 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:20.198583 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889780-qmbj9" Oct 30 19:00:20.198719 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:20.198582 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889780-qmbj9" event={"ID":"ef261723-21fe-493d-940f-394b9aca8fbc","Type":"ContainerDied","Data":"249f41d0d4f5f96d2593baa800117c7774a5b2d30809b1ed5698763e555fdd0c"} Oct 30 19:00:20.207988 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:20.207972 6960 scope.go:122] "RemoveContainer" containerID="8c3d75430323c7949dd1c24f31682738b9d380e72509285b64ac2f9b1c0980c4" Oct 30 19:00:20.224379 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:20.224358 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889780-qmbj9"] Oct 30 19:00:20.236126 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:20.236106 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-7a8fa2-29889780-qmbj9"] Oct 30 19:00:20.252333 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:20.252304 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ab7df0-29889780-z66fz"] Oct 30 19:00:20.262763 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:20.262723 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ab7df0-29889780-z66fz"] Oct 30 19:00:20.357860 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:20.357821 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:20.593741 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:20.593710 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0053dbc6-5623-498a-9fcf-9e89e1dc18ee" path="/var/lib/kubelet/pods/0053dbc6-5623-498a-9fcf-9e89e1dc18ee/volumes" Oct 30 19:00:20.594138 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:20.594125 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="734520a1-484d-4cc4-a340-302045e8ab82" path="/var/lib/kubelet/pods/734520a1-484d-4cc4-a340-302045e8ab82/volumes" Oct 30 19:00:20.594547 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:20.594534 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba9e1a42-4e4d-4b4a-8a2b-28823c69e663" path="/var/lib/kubelet/pods/ba9e1a42-4e4d-4b4a-8a2b-28823c69e663/volumes" Oct 30 19:00:20.595239 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:20.595225 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef261723-21fe-493d-940f-394b9aca8fbc" path="/var/lib/kubelet/pods/ef261723-21fe-493d-940f-394b9aca8fbc/volumes" Oct 30 19:00:21.454979 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:21.454948 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:22.423176 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:22.422610 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:23.419760 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:23.419726 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:23.628397 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:23.628366 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:23.932435 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:00:23.932396 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 19:00:24.573473 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:24.573443 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:25.409316 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:25.409283 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:26.669967 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:26.669936 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:27.698742 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:27.698704 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:28.743512 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:28.743477 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:28.854078 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:28.854026 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:29.753946 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:29.753909 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:30.609691 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:30.609655 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:31.589179 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:31.589104 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:31.610863 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:31.610833 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-743e0b/explicit-model-743e0b-kserve-78dbdb9b9c-tlgh4"] Oct 30 19:00:31.611545 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:31.611526 6960 cpu_manager.go:414] "containerMap: removing container" podUID="ba9e1a42-4e4d-4b4a-8a2b-28823c69e663" containerName="cleanup" Oct 30 19:00:31.611663 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:31.611545 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba9e1a42-4e4d-4b4a-8a2b-28823c69e663" containerName="cleanup" Oct 30 19:00:31.611663 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:31.611562 6960 cpu_manager.go:414] "containerMap: removing container" podUID="ba9e1a42-4e4d-4b4a-8a2b-28823c69e663" containerName="cleanup" Oct 30 19:00:31.611663 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:31.611569 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba9e1a42-4e4d-4b4a-8a2b-28823c69e663" containerName="cleanup" Oct 30 19:00:31.611663 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:31.611591 6960 cpu_manager.go:414] "containerMap: removing container" podUID="0053dbc6-5623-498a-9fcf-9e89e1dc18ee" containerName="cleanup" Oct 30 19:00:31.611663 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:31.611598 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0053dbc6-5623-498a-9fcf-9e89e1dc18ee" containerName="cleanup" Oct 30 19:00:31.611663 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:31.611606 6960 cpu_manager.go:414] "containerMap: removing container" podUID="ef261723-21fe-493d-940f-394b9aca8fbc" containerName="cleanup" Oct 30 19:00:31.611663 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:31.611613 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef261723-21fe-493d-940f-394b9aca8fbc" containerName="cleanup" Oct 30 19:00:31.611663 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:31.611638 6960 cpu_manager.go:414] "containerMap: removing container" podUID="0053dbc6-5623-498a-9fcf-9e89e1dc18ee" containerName="cleanup" Oct 30 19:00:31.611663 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:31.611646 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0053dbc6-5623-498a-9fcf-9e89e1dc18ee" containerName="cleanup" Oct 30 19:00:31.611663 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:31.611666 6960 cpu_manager.go:414] "containerMap: removing container" podUID="ba9e1a42-4e4d-4b4a-8a2b-28823c69e663" containerName="cleanup" Oct 30 19:00:31.612188 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:31.611673 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba9e1a42-4e4d-4b4a-8a2b-28823c69e663" containerName="cleanup" Oct 30 19:00:31.612188 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:31.611684 6960 cpu_manager.go:414] "containerMap: removing container" podUID="734520a1-484d-4cc4-a340-302045e8ab82" containerName="cleanup" Oct 30 19:00:31.612188 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:31.611691 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="734520a1-484d-4cc4-a340-302045e8ab82" containerName="cleanup" Oct 30 19:00:31.612188 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:31.611706 6960 cpu_manager.go:414] "containerMap: removing container" podUID="734520a1-484d-4cc4-a340-302045e8ab82" containerName="cleanup" Oct 30 19:00:31.612188 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:31.611713 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="734520a1-484d-4cc4-a340-302045e8ab82" containerName="cleanup" Oct 30 19:00:31.612188 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:31.611726 6960 cpu_manager.go:414] "containerMap: removing container" podUID="ef261723-21fe-493d-940f-394b9aca8fbc" containerName="cleanup" Oct 30 19:00:31.612188 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:31.611732 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef261723-21fe-493d-940f-394b9aca8fbc" containerName="cleanup" Oct 30 19:00:31.612188 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:31.611742 6960 cpu_manager.go:414] "containerMap: removing container" podUID="ef261723-21fe-493d-940f-394b9aca8fbc" containerName="cleanup" Oct 30 19:00:31.612188 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:31.611749 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef261723-21fe-493d-940f-394b9aca8fbc" containerName="cleanup" Oct 30 19:00:31.612188 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:31.611762 6960 cpu_manager.go:414] "containerMap: removing container" podUID="0053dbc6-5623-498a-9fcf-9e89e1dc18ee" containerName="cleanup" Oct 30 19:00:31.612188 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:31.611768 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0053dbc6-5623-498a-9fcf-9e89e1dc18ee" containerName="cleanup" Oct 30 19:00:31.612188 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:31.611794 6960 cpu_manager.go:414] "containerMap: removing container" podUID="734520a1-484d-4cc4-a340-302045e8ab82" containerName="cleanup" Oct 30 19:00:31.612188 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:31.611801 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="734520a1-484d-4cc4-a340-302045e8ab82" containerName="cleanup" Oct 30 19:00:31.612188 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:31.611980 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba9e1a42-4e4d-4b4a-8a2b-28823c69e663" containerName="cleanup" Oct 30 19:00:31.612188 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:31.611994 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="0053dbc6-5623-498a-9fcf-9e89e1dc18ee" containerName="cleanup" Oct 30 19:00:31.612188 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:31.612002 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef261723-21fe-493d-940f-394b9aca8fbc" containerName="cleanup" Oct 30 19:00:31.612188 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:31.612014 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="0053dbc6-5623-498a-9fcf-9e89e1dc18ee" containerName="cleanup" Oct 30 19:00:31.612188 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:31.612027 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="734520a1-484d-4cc4-a340-302045e8ab82" containerName="cleanup" Oct 30 19:00:31.612188 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:31.612036 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="734520a1-484d-4cc4-a340-302045e8ab82" containerName="cleanup" Oct 30 19:00:31.612188 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:31.612046 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef261723-21fe-493d-940f-394b9aca8fbc" containerName="cleanup" Oct 30 19:00:31.612188 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:31.612055 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="0053dbc6-5623-498a-9fcf-9e89e1dc18ee" containerName="cleanup" Oct 30 19:00:31.612188 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:31.612071 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba9e1a42-4e4d-4b4a-8a2b-28823c69e663" containerName="cleanup" Oct 30 19:00:31.617754 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:31.617732 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-743e0b/explicit-model-743e0b-kserve-78dbdb9b9c-tlgh4" Oct 30 19:00:31.634437 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:31.634411 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-743e0b\"/\"explicit-model-743e0b-kserve-self-signed-certs\"" Oct 30 19:00:31.634554 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:31.634515 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-743e0b\"/\"default-dockercfg-7qq6v\"" Oct 30 19:00:31.634554 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:31.634517 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-743e0b\"/\"openshift-service-ca.crt\"" Oct 30 19:00:31.634663 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:31.634637 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-743e0b/explicit-model-743e0b-kserve-78dbdb9b9c-tlgh4"] Oct 30 19:00:31.634663 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:31.634655 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-743e0b\"/\"kube-root-ca.crt\"" Oct 30 19:00:31.719757 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:31.719723 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91470ebb-aec0-4a9f-b5d3-44a82675d02c-home\") pod \"explicit-model-743e0b-kserve-78dbdb9b9c-tlgh4\" (UID: \"91470ebb-aec0-4a9f-b5d3-44a82675d02c\") " pod="ai-tenant-e2e-explicit-743e0b/explicit-model-743e0b-kserve-78dbdb9b9c-tlgh4" Oct 30 19:00:31.719965 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:31.719868 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91470ebb-aec0-4a9f-b5d3-44a82675d02c-tls-certs\") pod \"explicit-model-743e0b-kserve-78dbdb9b9c-tlgh4\" (UID: \"91470ebb-aec0-4a9f-b5d3-44a82675d02c\") " pod="ai-tenant-e2e-explicit-743e0b/explicit-model-743e0b-kserve-78dbdb9b9c-tlgh4" Oct 30 19:00:31.720041 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:31.720017 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88qp5\" (UniqueName: \"kubernetes.io/projected/91470ebb-aec0-4a9f-b5d3-44a82675d02c-kube-api-access-88qp5\") pod \"explicit-model-743e0b-kserve-78dbdb9b9c-tlgh4\" (UID: \"91470ebb-aec0-4a9f-b5d3-44a82675d02c\") " pod="ai-tenant-e2e-explicit-743e0b/explicit-model-743e0b-kserve-78dbdb9b9c-tlgh4" Oct 30 19:00:31.720099 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:31.720084 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91470ebb-aec0-4a9f-b5d3-44a82675d02c-tmp-dir\") pod \"explicit-model-743e0b-kserve-78dbdb9b9c-tlgh4\" (UID: \"91470ebb-aec0-4a9f-b5d3-44a82675d02c\") " pod="ai-tenant-e2e-explicit-743e0b/explicit-model-743e0b-kserve-78dbdb9b9c-tlgh4" Oct 30 19:00:31.720238 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:31.720219 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91470ebb-aec0-4a9f-b5d3-44a82675d02c-model-cache\") pod \"explicit-model-743e0b-kserve-78dbdb9b9c-tlgh4\" (UID: \"91470ebb-aec0-4a9f-b5d3-44a82675d02c\") " pod="ai-tenant-e2e-explicit-743e0b/explicit-model-743e0b-kserve-78dbdb9b9c-tlgh4" Oct 30 19:00:31.720516 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:31.720491 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91470ebb-aec0-4a9f-b5d3-44a82675d02c-dshm\") pod \"explicit-model-743e0b-kserve-78dbdb9b9c-tlgh4\" (UID: \"91470ebb-aec0-4a9f-b5d3-44a82675d02c\") " pod="ai-tenant-e2e-explicit-743e0b/explicit-model-743e0b-kserve-78dbdb9b9c-tlgh4" Oct 30 19:00:31.769299 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:31.769268 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:31.821857 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:31.821822 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88qp5\" (UniqueName: \"kubernetes.io/projected/91470ebb-aec0-4a9f-b5d3-44a82675d02c-kube-api-access-88qp5\") pod \"explicit-model-743e0b-kserve-78dbdb9b9c-tlgh4\" (UID: \"91470ebb-aec0-4a9f-b5d3-44a82675d02c\") " pod="ai-tenant-e2e-explicit-743e0b/explicit-model-743e0b-kserve-78dbdb9b9c-tlgh4" Oct 30 19:00:31.822044 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:31.821872 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91470ebb-aec0-4a9f-b5d3-44a82675d02c-tmp-dir\") pod \"explicit-model-743e0b-kserve-78dbdb9b9c-tlgh4\" (UID: \"91470ebb-aec0-4a9f-b5d3-44a82675d02c\") " pod="ai-tenant-e2e-explicit-743e0b/explicit-model-743e0b-kserve-78dbdb9b9c-tlgh4" Oct 30 19:00:31.822044 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:31.821925 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91470ebb-aec0-4a9f-b5d3-44a82675d02c-model-cache\") pod \"explicit-model-743e0b-kserve-78dbdb9b9c-tlgh4\" (UID: \"91470ebb-aec0-4a9f-b5d3-44a82675d02c\") " pod="ai-tenant-e2e-explicit-743e0b/explicit-model-743e0b-kserve-78dbdb9b9c-tlgh4" Oct 30 19:00:31.822185 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:31.822064 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91470ebb-aec0-4a9f-b5d3-44a82675d02c-dshm\") pod \"explicit-model-743e0b-kserve-78dbdb9b9c-tlgh4\" (UID: \"91470ebb-aec0-4a9f-b5d3-44a82675d02c\") " pod="ai-tenant-e2e-explicit-743e0b/explicit-model-743e0b-kserve-78dbdb9b9c-tlgh4" Oct 30 19:00:31.822185 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:31.822096 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91470ebb-aec0-4a9f-b5d3-44a82675d02c-home\") pod \"explicit-model-743e0b-kserve-78dbdb9b9c-tlgh4\" (UID: \"91470ebb-aec0-4a9f-b5d3-44a82675d02c\") " pod="ai-tenant-e2e-explicit-743e0b/explicit-model-743e0b-kserve-78dbdb9b9c-tlgh4" Oct 30 19:00:31.822295 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:31.822212 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91470ebb-aec0-4a9f-b5d3-44a82675d02c-tls-certs\") pod \"explicit-model-743e0b-kserve-78dbdb9b9c-tlgh4\" (UID: \"91470ebb-aec0-4a9f-b5d3-44a82675d02c\") " pod="ai-tenant-e2e-explicit-743e0b/explicit-model-743e0b-kserve-78dbdb9b9c-tlgh4" Oct 30 19:00:31.822380 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:31.822345 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91470ebb-aec0-4a9f-b5d3-44a82675d02c-tmp-dir\") pod \"explicit-model-743e0b-kserve-78dbdb9b9c-tlgh4\" (UID: \"91470ebb-aec0-4a9f-b5d3-44a82675d02c\") " pod="ai-tenant-e2e-explicit-743e0b/explicit-model-743e0b-kserve-78dbdb9b9c-tlgh4" Oct 30 19:00:31.822520 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:31.822495 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91470ebb-aec0-4a9f-b5d3-44a82675d02c-home\") pod \"explicit-model-743e0b-kserve-78dbdb9b9c-tlgh4\" (UID: \"91470ebb-aec0-4a9f-b5d3-44a82675d02c\") " pod="ai-tenant-e2e-explicit-743e0b/explicit-model-743e0b-kserve-78dbdb9b9c-tlgh4" Oct 30 19:00:31.822737 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:31.822718 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91470ebb-aec0-4a9f-b5d3-44a82675d02c-model-cache\") pod \"explicit-model-743e0b-kserve-78dbdb9b9c-tlgh4\" (UID: \"91470ebb-aec0-4a9f-b5d3-44a82675d02c\") " pod="ai-tenant-e2e-explicit-743e0b/explicit-model-743e0b-kserve-78dbdb9b9c-tlgh4" Oct 30 19:00:31.825078 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:31.825051 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91470ebb-aec0-4a9f-b5d3-44a82675d02c-tls-certs\") pod \"explicit-model-743e0b-kserve-78dbdb9b9c-tlgh4\" (UID: \"91470ebb-aec0-4a9f-b5d3-44a82675d02c\") " pod="ai-tenant-e2e-explicit-743e0b/explicit-model-743e0b-kserve-78dbdb9b9c-tlgh4" Oct 30 19:00:31.825248 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:31.825227 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91470ebb-aec0-4a9f-b5d3-44a82675d02c-dshm\") pod \"explicit-model-743e0b-kserve-78dbdb9b9c-tlgh4\" (UID: \"91470ebb-aec0-4a9f-b5d3-44a82675d02c\") " pod="ai-tenant-e2e-explicit-743e0b/explicit-model-743e0b-kserve-78dbdb9b9c-tlgh4" Oct 30 19:00:31.834449 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:31.834426 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88qp5\" (UniqueName: \"kubernetes.io/projected/91470ebb-aec0-4a9f-b5d3-44a82675d02c-kube-api-access-88qp5\") pod \"explicit-model-743e0b-kserve-78dbdb9b9c-tlgh4\" (UID: \"91470ebb-aec0-4a9f-b5d3-44a82675d02c\") " pod="ai-tenant-e2e-explicit-743e0b/explicit-model-743e0b-kserve-78dbdb9b9c-tlgh4" Oct 30 19:00:31.930222 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:31.929953 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-743e0b/explicit-model-743e0b-kserve-78dbdb9b9c-tlgh4" Oct 30 19:00:32.334041 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:32.334011 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-743e0b/explicit-model-743e0b-kserve-78dbdb9b9c-tlgh4"] Oct 30 19:00:32.334700 ip-10-0-140-163 kubenswrapper[6960]: W1030 19:00:32.334674 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod91470ebb_aec0_4a9f_b5d3_44a82675d02c.slice/crio-a3e34159a7e26d048f2e540431bae73544e1c38a4e1fae8a9396eb232c9f894a WatchSource:0}: Error finding container a3e34159a7e26d048f2e540431bae73544e1c38a4e1fae8a9396eb232c9f894a: Status 404 returned error can't find the container with id a3e34159a7e26d048f2e540431bae73544e1c38a4e1fae8a9396eb232c9f894a Oct 30 19:00:33.163312 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:33.163263 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:33.315550 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:33.315515 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-743e0b/explicit-model-743e0b-kserve-78dbdb9b9c-tlgh4" event={"ID":"91470ebb-aec0-4a9f-b5d3-44a82675d02c","Type":"ContainerStarted","Data":"398480d49483db69d4222440b23380babadad7fa5f720a214d0e7692a3b527de"} Oct 30 19:00:33.315722 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:33.315564 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-743e0b/explicit-model-743e0b-kserve-78dbdb9b9c-tlgh4" event={"ID":"91470ebb-aec0-4a9f-b5d3-44a82675d02c","Type":"ContainerStarted","Data":"a3e34159a7e26d048f2e540431bae73544e1c38a4e1fae8a9396eb232c9f894a"} Oct 30 19:00:33.341082 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:33.341029 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-743e0b/explicit-model-743e0b-kserve-78dbdb9b9c-tlgh4" podStartSLOduration=2.341015364 podStartE2EDuration="2.341015364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:33.336483237 +0000 UTC m=+1600.865514375" watchObservedRunningTime="2026-10-30 19:00:33.341015364 +0000 UTC m=+1600.870046502" Oct 30 19:00:33.930744 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:33.930701 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-743e0b/explicit-model-743e0b-kserve-78dbdb9b9c-tlgh4" Oct 30 19:00:34.211855 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:34.211758 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:35.055845 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:35.055808 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:36.068777 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:36.068733 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:36.781098 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:36.781061 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-kmw8d"] Oct 30 19:00:36.782424 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:36.782399 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef261723-21fe-493d-940f-394b9aca8fbc" containerName="cleanup" Oct 30 19:00:36.782557 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:36.782436 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="734520a1-484d-4cc4-a340-302045e8ab82" containerName="cleanup" Oct 30 19:00:36.782557 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:36.782453 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba9e1a42-4e4d-4b4a-8a2b-28823c69e663" containerName="cleanup" Oct 30 19:00:36.786710 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:36.786687 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-kmw8d" Oct 30 19:00:36.811657 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:36.811614 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-kmw8d"] Oct 30 19:00:36.837227 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:36.837130 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:36.878359 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:36.878320 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-784fbf-openshift-default-dockercfg-xjmtj\"" Oct 30 19:00:36.909794 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:36.909762 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b416cfb-588b-4dde-bd77-9d5fca225a87-workload-socket\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-kmw8d\" (UID: \"7b416cfb-588b-4dde-bd77-9d5fca225a87\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-kmw8d" Oct 30 19:00:36.909975 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:36.909809 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b416cfb-588b-4dde-bd77-9d5fca225a87-workload-certs\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-kmw8d\" (UID: \"7b416cfb-588b-4dde-bd77-9d5fca225a87\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-kmw8d" Oct 30 19:00:36.910049 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:36.910021 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b416cfb-588b-4dde-bd77-9d5fca225a87-istio-envoy\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-kmw8d\" (UID: \"7b416cfb-588b-4dde-bd77-9d5fca225a87\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-kmw8d" Oct 30 19:00:36.910284 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:36.910265 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b416cfb-588b-4dde-bd77-9d5fca225a87-credential-socket\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-kmw8d\" (UID: \"7b416cfb-588b-4dde-bd77-9d5fca225a87\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-kmw8d" Oct 30 19:00:36.910373 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:36.910300 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b416cfb-588b-4dde-bd77-9d5fca225a87-istio-token\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-kmw8d\" (UID: \"7b416cfb-588b-4dde-bd77-9d5fca225a87\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-kmw8d" Oct 30 19:00:36.910373 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:36.910350 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b416cfb-588b-4dde-bd77-9d5fca225a87-istiod-ca-cert\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-kmw8d\" (UID: \"7b416cfb-588b-4dde-bd77-9d5fca225a87\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-kmw8d" Oct 30 19:00:36.910453 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:36.910381 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b416cfb-588b-4dde-bd77-9d5fca225a87-istio-data\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-kmw8d\" (UID: \"7b416cfb-588b-4dde-bd77-9d5fca225a87\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-kmw8d" Oct 30 19:00:36.910453 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:36.910435 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrfl8\" (UniqueName: \"kubernetes.io/projected/7b416cfb-588b-4dde-bd77-9d5fca225a87-kube-api-access-hrfl8\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-kmw8d\" (UID: \"7b416cfb-588b-4dde-bd77-9d5fca225a87\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-kmw8d" Oct 30 19:00:36.910527 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:36.910497 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b416cfb-588b-4dde-bd77-9d5fca225a87-istio-podinfo\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-kmw8d\" (UID: \"7b416cfb-588b-4dde-bd77-9d5fca225a87\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-kmw8d" Oct 30 19:00:37.011799 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:37.011767 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b416cfb-588b-4dde-bd77-9d5fca225a87-istio-podinfo\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-kmw8d\" (UID: \"7b416cfb-588b-4dde-bd77-9d5fca225a87\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-kmw8d" Oct 30 19:00:37.011953 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:37.011838 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b416cfb-588b-4dde-bd77-9d5fca225a87-workload-socket\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-kmw8d\" (UID: \"7b416cfb-588b-4dde-bd77-9d5fca225a87\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-kmw8d" Oct 30 19:00:37.011953 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:37.011861 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b416cfb-588b-4dde-bd77-9d5fca225a87-workload-certs\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-kmw8d\" (UID: \"7b416cfb-588b-4dde-bd77-9d5fca225a87\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-kmw8d" Oct 30 19:00:37.011953 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:37.011914 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b416cfb-588b-4dde-bd77-9d5fca225a87-istio-envoy\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-kmw8d\" (UID: \"7b416cfb-588b-4dde-bd77-9d5fca225a87\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-kmw8d" Oct 30 19:00:37.012093 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:37.012073 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b416cfb-588b-4dde-bd77-9d5fca225a87-credential-socket\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-kmw8d\" (UID: \"7b416cfb-588b-4dde-bd77-9d5fca225a87\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-kmw8d" Oct 30 19:00:37.012133 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:37.012098 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b416cfb-588b-4dde-bd77-9d5fca225a87-istio-token\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-kmw8d\" (UID: \"7b416cfb-588b-4dde-bd77-9d5fca225a87\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-kmw8d" Oct 30 19:00:37.012205 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:37.012168 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b416cfb-588b-4dde-bd77-9d5fca225a87-istiod-ca-cert\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-kmw8d\" (UID: \"7b416cfb-588b-4dde-bd77-9d5fca225a87\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-kmw8d" Oct 30 19:00:37.012261 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:37.012206 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b416cfb-588b-4dde-bd77-9d5fca225a87-istio-data\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-kmw8d\" (UID: \"7b416cfb-588b-4dde-bd77-9d5fca225a87\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-kmw8d" Oct 30 19:00:37.012323 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:37.012264 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrfl8\" (UniqueName: \"kubernetes.io/projected/7b416cfb-588b-4dde-bd77-9d5fca225a87-kube-api-access-hrfl8\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-kmw8d\" (UID: \"7b416cfb-588b-4dde-bd77-9d5fca225a87\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-kmw8d" Oct 30 19:00:37.012383 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:37.012329 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b416cfb-588b-4dde-bd77-9d5fca225a87-workload-certs\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-kmw8d\" (UID: \"7b416cfb-588b-4dde-bd77-9d5fca225a87\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-kmw8d" Oct 30 19:00:37.012572 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:37.012548 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b416cfb-588b-4dde-bd77-9d5fca225a87-workload-socket\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-kmw8d\" (UID: \"7b416cfb-588b-4dde-bd77-9d5fca225a87\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-kmw8d" Oct 30 19:00:37.012875 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:37.012596 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b416cfb-588b-4dde-bd77-9d5fca225a87-istio-data\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-kmw8d\" (UID: \"7b416cfb-588b-4dde-bd77-9d5fca225a87\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-kmw8d" Oct 30 19:00:37.012875 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:37.012677 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b416cfb-588b-4dde-bd77-9d5fca225a87-credential-socket\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-kmw8d\" (UID: \"7b416cfb-588b-4dde-bd77-9d5fca225a87\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-kmw8d" Oct 30 19:00:37.012875 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:37.012819 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b416cfb-588b-4dde-bd77-9d5fca225a87-istiod-ca-cert\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-kmw8d\" (UID: \"7b416cfb-588b-4dde-bd77-9d5fca225a87\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-kmw8d" Oct 30 19:00:37.014408 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:37.014390 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b416cfb-588b-4dde-bd77-9d5fca225a87-istio-envoy\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-kmw8d\" (UID: \"7b416cfb-588b-4dde-bd77-9d5fca225a87\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-kmw8d" Oct 30 19:00:37.014561 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:37.014540 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b416cfb-588b-4dde-bd77-9d5fca225a87-istio-podinfo\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-kmw8d\" (UID: \"7b416cfb-588b-4dde-bd77-9d5fca225a87\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-kmw8d" Oct 30 19:00:37.027028 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:37.027005 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b416cfb-588b-4dde-bd77-9d5fca225a87-istio-token\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-kmw8d\" (UID: \"7b416cfb-588b-4dde-bd77-9d5fca225a87\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-kmw8d" Oct 30 19:00:37.027138 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:37.027118 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrfl8\" (UniqueName: \"kubernetes.io/projected/7b416cfb-588b-4dde-bd77-9d5fca225a87-kube-api-access-hrfl8\") pod \"e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-kmw8d\" (UID: \"7b416cfb-588b-4dde-bd77-9d5fca225a87\") " pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-kmw8d" Oct 30 19:00:37.106389 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:37.106358 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-kmw8d" Oct 30 19:00:37.402578 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:37.402553 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-kmw8d"] Oct 30 19:00:37.402902 ip-10-0-140-163 kubenswrapper[6960]: W1030 19:00:37.402877 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b416cfb_588b_4dde_bd77_9d5fca225a87.slice/crio-50812907dadcc5325cdfa094fd41eb30917ed82a3f3d6bf90392279e81bcc8cd WatchSource:0}: Error finding container 50812907dadcc5325cdfa094fd41eb30917ed82a3f3d6bf90392279e81bcc8cd: Status 404 returned error can't find the container with id 50812907dadcc5325cdfa094fd41eb30917ed82a3f3d6bf90392279e81bcc8cd Oct 30 19:00:37.405995 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:37.405962 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 19:00:37.406079 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:37.406052 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 19:00:37.406126 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:37.406081 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 19:00:37.988208 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:37.988121 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:38.351952 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:38.351910 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-kmw8d" event={"ID":"7b416cfb-588b-4dde-bd77-9d5fca225a87","Type":"ContainerStarted","Data":"36681657a405f967bda0f6142f8f4a3f6158ffc703f0dd4ee1ad969b96fb4877"} Oct 30 19:00:38.352383 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:38.351966 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-kmw8d" event={"ID":"7b416cfb-588b-4dde-bd77-9d5fca225a87","Type":"ContainerStarted","Data":"50812907dadcc5325cdfa094fd41eb30917ed82a3f3d6bf90392279e81bcc8cd"} Oct 30 19:00:38.407663 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:38.407608 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-kmw8d" podStartSLOduration=2.4075899 podStartE2EDuration="2.4075899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:38.406399058 +0000 UTC m=+1605.935430202" watchObservedRunningTime="2026-10-30 19:00:38.4075899 +0000 UTC m=+1605.936621038" Oct 30 19:00:39.087172 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:39.082759 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:39.107396 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:39.107371 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-kmw8d" Oct 30 19:00:39.109251 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:39.109220 6960 patch_prober.go:29] interesting pod/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-kmw8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 30 19:00:39.109413 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:39.109282 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-kmw8d" podUID="7b416cfb-588b-4dde-bd77-9d5fca225a87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 30 19:00:39.971127 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:39.971094 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:40.106849 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:40.106813 6960 patch_prober.go:29] interesting pod/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-kmw8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 30 19:00:40.107021 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:40.106868 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-kmw8d" podUID="7b416cfb-588b-4dde-bd77-9d5fca225a87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 30 19:00:40.920515 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:40.920481 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:41.106802 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:41.106767 6960 patch_prober.go:29] interesting pod/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-kmw8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 30 19:00:41.107300 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:41.106830 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-kmw8d" podUID="7b416cfb-588b-4dde-bd77-9d5fca225a87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 30 19:00:41.783039 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:41.782999 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:41.930863 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:41.930826 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-743e0b/explicit-model-743e0b-kserve-78dbdb9b9c-tlgh4" Oct 30 19:00:41.943216 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:41.943189 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-743e0b/explicit-model-743e0b-kserve-78dbdb9b9c-tlgh4" Oct 30 19:00:42.107669 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:42.107635 6960 patch_prober.go:29] interesting pod/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-kmw8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 30 19:00:42.108066 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:42.107687 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-kmw8d" podUID="7b416cfb-588b-4dde-bd77-9d5fca225a87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 30 19:00:42.416834 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:42.416759 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-743e0b/explicit-model-743e0b-kserve-78dbdb9b9c-tlgh4" Oct 30 19:00:42.789519 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:42.789417 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:43.107343 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:43.107312 6960 patch_prober.go:29] interesting pod/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-kmw8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 30 19:00:43.107516 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:43.107387 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-kmw8d" podUID="7b416cfb-588b-4dde-bd77-9d5fca225a87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 30 19:00:43.885418 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:43.885386 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-743e0b/explicit-model-743e0b-kserve-78dbdb9b9c-tlgh4"] Oct 30 19:00:43.945108 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:43.945075 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:44.117816 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:44.117767 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-kmw8d" Oct 30 19:00:44.118931 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:44.118910 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-kmw8d" Oct 30 19:00:44.120156 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:44.120130 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-kmw8d" Oct 30 19:00:44.241343 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:44.241257 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:45.254285 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:45.254251 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:45.445608 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:45.445570 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:46.150383 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:46.150350 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:47.035099 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:47.035064 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:47.744590 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:47.744552 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:48.052727 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:48.052646 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:49.036916 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:49.036884 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:49.134786 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:49.134754 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-m7qgw"] Oct 30 19:00:49.140333 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:49.140309 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-m7qgw" Oct 30 19:00:49.144191 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:49.144169 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-743e0b-openshift-default-dockercfg-dntsq\"" Oct 30 19:00:49.164196 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:49.164171 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-m7qgw"] Oct 30 19:00:49.179643 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:49.179610 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbxzw\" (UniqueName: \"kubernetes.io/projected/5fcd353e-568f-4a3e-b1c6-1f44f45f5e57-kube-api-access-sbxzw\") pod \"e2e-explicit-743e0b-openshift-default-65cdbd6b45-m7qgw\" (UID: \"5fcd353e-568f-4a3e-b1c6-1f44f45f5e57\") " pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-m7qgw" Oct 30 19:00:49.179819 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:49.179671 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fcd353e-568f-4a3e-b1c6-1f44f45f5e57-istio-token\") pod \"e2e-explicit-743e0b-openshift-default-65cdbd6b45-m7qgw\" (UID: \"5fcd353e-568f-4a3e-b1c6-1f44f45f5e57\") " pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-m7qgw" Oct 30 19:00:49.179819 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:49.179707 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fcd353e-568f-4a3e-b1c6-1f44f45f5e57-credential-socket\") pod \"e2e-explicit-743e0b-openshift-default-65cdbd6b45-m7qgw\" (UID: \"5fcd353e-568f-4a3e-b1c6-1f44f45f5e57\") " pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-m7qgw" Oct 30 19:00:49.179819 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:49.179748 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fcd353e-568f-4a3e-b1c6-1f44f45f5e57-istio-data\") pod \"e2e-explicit-743e0b-openshift-default-65cdbd6b45-m7qgw\" (UID: \"5fcd353e-568f-4a3e-b1c6-1f44f45f5e57\") " pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-m7qgw" Oct 30 19:00:49.180010 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:49.179832 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fcd353e-568f-4a3e-b1c6-1f44f45f5e57-istio-envoy\") pod \"e2e-explicit-743e0b-openshift-default-65cdbd6b45-m7qgw\" (UID: \"5fcd353e-568f-4a3e-b1c6-1f44f45f5e57\") " pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-m7qgw" Oct 30 19:00:49.180010 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:49.179877 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fcd353e-568f-4a3e-b1c6-1f44f45f5e57-workload-certs\") pod \"e2e-explicit-743e0b-openshift-default-65cdbd6b45-m7qgw\" (UID: \"5fcd353e-568f-4a3e-b1c6-1f44f45f5e57\") " pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-m7qgw" Oct 30 19:00:49.180010 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:49.179915 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fcd353e-568f-4a3e-b1c6-1f44f45f5e57-istiod-ca-cert\") pod \"e2e-explicit-743e0b-openshift-default-65cdbd6b45-m7qgw\" (UID: \"5fcd353e-568f-4a3e-b1c6-1f44f45f5e57\") " pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-m7qgw" Oct 30 19:00:49.180135 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:49.180034 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fcd353e-568f-4a3e-b1c6-1f44f45f5e57-istio-podinfo\") pod \"e2e-explicit-743e0b-openshift-default-65cdbd6b45-m7qgw\" (UID: \"5fcd353e-568f-4a3e-b1c6-1f44f45f5e57\") " pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-m7qgw" Oct 30 19:00:49.180135 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:49.180086 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fcd353e-568f-4a3e-b1c6-1f44f45f5e57-workload-socket\") pod \"e2e-explicit-743e0b-openshift-default-65cdbd6b45-m7qgw\" (UID: \"5fcd353e-568f-4a3e-b1c6-1f44f45f5e57\") " pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-m7qgw" Oct 30 19:00:49.281991 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:49.281963 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbxzw\" (UniqueName: \"kubernetes.io/projected/5fcd353e-568f-4a3e-b1c6-1f44f45f5e57-kube-api-access-sbxzw\") pod \"e2e-explicit-743e0b-openshift-default-65cdbd6b45-m7qgw\" (UID: \"5fcd353e-568f-4a3e-b1c6-1f44f45f5e57\") " pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-m7qgw" Oct 30 19:00:49.282198 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:49.282001 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fcd353e-568f-4a3e-b1c6-1f44f45f5e57-istio-token\") pod \"e2e-explicit-743e0b-openshift-default-65cdbd6b45-m7qgw\" (UID: \"5fcd353e-568f-4a3e-b1c6-1f44f45f5e57\") " pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-m7qgw" Oct 30 19:00:49.282198 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:49.282029 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fcd353e-568f-4a3e-b1c6-1f44f45f5e57-credential-socket\") pod \"e2e-explicit-743e0b-openshift-default-65cdbd6b45-m7qgw\" (UID: \"5fcd353e-568f-4a3e-b1c6-1f44f45f5e57\") " pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-m7qgw" Oct 30 19:00:49.282198 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:49.282064 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fcd353e-568f-4a3e-b1c6-1f44f45f5e57-istio-data\") pod \"e2e-explicit-743e0b-openshift-default-65cdbd6b45-m7qgw\" (UID: \"5fcd353e-568f-4a3e-b1c6-1f44f45f5e57\") " pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-m7qgw" Oct 30 19:00:49.282198 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:49.282131 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fcd353e-568f-4a3e-b1c6-1f44f45f5e57-istio-envoy\") pod \"e2e-explicit-743e0b-openshift-default-65cdbd6b45-m7qgw\" (UID: \"5fcd353e-568f-4a3e-b1c6-1f44f45f5e57\") " pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-m7qgw" Oct 30 19:00:49.282198 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:49.282189 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fcd353e-568f-4a3e-b1c6-1f44f45f5e57-workload-certs\") pod \"e2e-explicit-743e0b-openshift-default-65cdbd6b45-m7qgw\" (UID: \"5fcd353e-568f-4a3e-b1c6-1f44f45f5e57\") " pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-m7qgw" Oct 30 19:00:49.282513 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:49.282224 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fcd353e-568f-4a3e-b1c6-1f44f45f5e57-istiod-ca-cert\") pod \"e2e-explicit-743e0b-openshift-default-65cdbd6b45-m7qgw\" (UID: \"5fcd353e-568f-4a3e-b1c6-1f44f45f5e57\") " pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-m7qgw" Oct 30 19:00:49.282513 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:49.282296 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fcd353e-568f-4a3e-b1c6-1f44f45f5e57-istio-podinfo\") pod \"e2e-explicit-743e0b-openshift-default-65cdbd6b45-m7qgw\" (UID: \"5fcd353e-568f-4a3e-b1c6-1f44f45f5e57\") " pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-m7qgw" Oct 30 19:00:49.282513 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:49.282329 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fcd353e-568f-4a3e-b1c6-1f44f45f5e57-workload-socket\") pod \"e2e-explicit-743e0b-openshift-default-65cdbd6b45-m7qgw\" (UID: \"5fcd353e-568f-4a3e-b1c6-1f44f45f5e57\") " pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-m7qgw" Oct 30 19:00:49.282650 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:49.282554 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fcd353e-568f-4a3e-b1c6-1f44f45f5e57-credential-socket\") pod \"e2e-explicit-743e0b-openshift-default-65cdbd6b45-m7qgw\" (UID: \"5fcd353e-568f-4a3e-b1c6-1f44f45f5e57\") " pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-m7qgw" Oct 30 19:00:49.282650 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:49.282637 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fcd353e-568f-4a3e-b1c6-1f44f45f5e57-istio-data\") pod \"e2e-explicit-743e0b-openshift-default-65cdbd6b45-m7qgw\" (UID: \"5fcd353e-568f-4a3e-b1c6-1f44f45f5e57\") " pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-m7qgw" Oct 30 19:00:49.282724 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:49.282683 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fcd353e-568f-4a3e-b1c6-1f44f45f5e57-workload-certs\") pod \"e2e-explicit-743e0b-openshift-default-65cdbd6b45-m7qgw\" (UID: \"5fcd353e-568f-4a3e-b1c6-1f44f45f5e57\") " pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-m7qgw" Oct 30 19:00:49.282950 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:49.282930 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fcd353e-568f-4a3e-b1c6-1f44f45f5e57-istiod-ca-cert\") pod \"e2e-explicit-743e0b-openshift-default-65cdbd6b45-m7qgw\" (UID: \"5fcd353e-568f-4a3e-b1c6-1f44f45f5e57\") " pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-m7qgw" Oct 30 19:00:49.283179 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:49.283139 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fcd353e-568f-4a3e-b1c6-1f44f45f5e57-workload-socket\") pod \"e2e-explicit-743e0b-openshift-default-65cdbd6b45-m7qgw\" (UID: \"5fcd353e-568f-4a3e-b1c6-1f44f45f5e57\") " pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-m7qgw" Oct 30 19:00:49.284587 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:49.284567 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fcd353e-568f-4a3e-b1c6-1f44f45f5e57-istio-envoy\") pod \"e2e-explicit-743e0b-openshift-default-65cdbd6b45-m7qgw\" (UID: \"5fcd353e-568f-4a3e-b1c6-1f44f45f5e57\") " pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-m7qgw" Oct 30 19:00:49.284921 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:49.284902 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fcd353e-568f-4a3e-b1c6-1f44f45f5e57-istio-podinfo\") pod \"e2e-explicit-743e0b-openshift-default-65cdbd6b45-m7qgw\" (UID: \"5fcd353e-568f-4a3e-b1c6-1f44f45f5e57\") " pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-m7qgw" Oct 30 19:00:49.293988 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:49.293933 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbxzw\" (UniqueName: \"kubernetes.io/projected/5fcd353e-568f-4a3e-b1c6-1f44f45f5e57-kube-api-access-sbxzw\") pod \"e2e-explicit-743e0b-openshift-default-65cdbd6b45-m7qgw\" (UID: \"5fcd353e-568f-4a3e-b1c6-1f44f45f5e57\") " pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-m7qgw" Oct 30 19:00:49.294557 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:49.294541 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fcd353e-568f-4a3e-b1c6-1f44f45f5e57-istio-token\") pod \"e2e-explicit-743e0b-openshift-default-65cdbd6b45-m7qgw\" (UID: \"5fcd353e-568f-4a3e-b1c6-1f44f45f5e57\") " pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-m7qgw" Oct 30 19:00:49.468894 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:49.468868 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-m7qgw" Oct 30 19:00:49.761122 ip-10-0-140-163 kubenswrapper[6960]: W1030 19:00:49.761076 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fcd353e_568f_4a3e_b1c6_1f44f45f5e57.slice/crio-b89888d28ece8688f882c4e61eeb63b8f8171751687ed062385bd30765df003e WatchSource:0}: Error finding container b89888d28ece8688f882c4e61eeb63b8f8171751687ed062385bd30765df003e: Status 404 returned error can't find the container with id b89888d28ece8688f882c4e61eeb63b8f8171751687ed062385bd30765df003e Oct 30 19:00:49.761289 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:49.761172 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-m7qgw"] Oct 30 19:00:49.764504 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:49.764470 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 19:00:49.764588 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:49.764552 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 19:00:49.764627 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:49.764596 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 19:00:49.962540 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:49.962483 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:50.486003 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:50.485964 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-m7qgw" event={"ID":"5fcd353e-568f-4a3e-b1c6-1f44f45f5e57","Type":"ContainerStarted","Data":"4c767240e4fa49b29dd6e271eea3afab0088fbda81277e0ec5d61dcfa3b7b531"} Oct 30 19:00:50.486472 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:50.486031 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-m7qgw" event={"ID":"5fcd353e-568f-4a3e-b1c6-1f44f45f5e57","Type":"ContainerStarted","Data":"b89888d28ece8688f882c4e61eeb63b8f8171751687ed062385bd30765df003e"} Oct 30 19:00:50.520457 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:50.520380 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-m7qgw" podStartSLOduration=1.5203610570000001 podStartE2EDuration="1.520361057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:50.508645148 +0000 UTC m=+1618.037676286" watchObservedRunningTime="2026-10-30 19:00:50.520361057 +0000 UTC m=+1618.049392195" Oct 30 19:00:51.081352 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:51.081315 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:51.472555 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:51.472475 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-m7qgw" Oct 30 19:00:51.479030 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:51.478997 6960 patch_prober.go:29] interesting pod/e2e-explicit-743e0b-openshift-default-65cdbd6b45-m7qgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 30 19:00:51.479272 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:51.479246 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-m7qgw" podUID="5fcd353e-568f-4a3e-b1c6-1f44f45f5e57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 30 19:00:52.117328 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:52.117275 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:52.333697 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:52.333663 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:52.469570 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:52.469490 6960 patch_prober.go:29] interesting pod/e2e-explicit-743e0b-openshift-default-65cdbd6b45-m7qgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 30 19:00:52.469828 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:52.469800 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-m7qgw" podUID="5fcd353e-568f-4a3e-b1c6-1f44f45f5e57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 30 19:00:52.985322 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:52.985287 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:53.474171 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:53.472500 6960 patch_prober.go:29] interesting pod/e2e-explicit-743e0b-openshift-default-65cdbd6b45-m7qgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 30 19:00:53.474171 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:53.472558 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-m7qgw" podUID="5fcd353e-568f-4a3e-b1c6-1f44f45f5e57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 30 19:00:53.852841 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:53.852805 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:53.998421 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:00:53.998385 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 19:00:54.403850 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:54.403822 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-743e0b-85c6498fc5-hdpjx"] Oct 30 19:00:54.438953 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:54.438919 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:54.480203 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:54.471645 6960 patch_prober.go:29] interesting pod/e2e-explicit-743e0b-openshift-default-65cdbd6b45-m7qgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 30 19:00:54.480203 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:54.471698 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-m7qgw" podUID="5fcd353e-568f-4a3e-b1c6-1f44f45f5e57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 30 19:00:54.580400 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:54.580360 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:55.138883 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:55.138839 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:55.479817 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:55.479732 6960 patch_prober.go:29] interesting pod/e2e-explicit-743e0b-openshift-default-65cdbd6b45-m7qgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 30 19:00:55.480107 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:55.480066 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-m7qgw" podUID="5fcd353e-568f-4a3e-b1c6-1f44f45f5e57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 30 19:00:56.009994 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:56.009932 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:56.474322 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:56.474284 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-m7qgw" Oct 30 19:00:56.474727 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:56.474701 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-m7qgw" Oct 30 19:00:56.475402 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:56.475385 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-m7qgw" Oct 30 19:00:57.264634 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:57.264535 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:58.122648 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:58.122618 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:59.007873 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:59.007840 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:59.404622 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:59.404579 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-743e0b-85c6498fc5-hdpjx" podUID="4b1ff70c-fcdf-4cdc-904c-37cd17e95ded" containerName="payload-pre-processing" containerID="cri-o://05355ffe6184f3946e526eb1fc932f2c3b6a5648acc42bb6e7aa1a059324f027" gracePeriod=25 Oct 30 19:00:59.441752 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:59.441699 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-743e0b/explicit-model-743e0b-kserve-78dbdb9b9c-tlgh4" podUID="91470ebb-aec0-4a9f-b5d3-44a82675d02c" containerName="main" containerID="cri-o://398480d49483db69d4222440b23380babadad7fa5f720a214d0e7692a3b527de" gracePeriod=45 Oct 30 19:00:59.556953 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:59.556922 6960 generic.go:356] "Generic (PLEG): container finished" podID="4b1ff70c-fcdf-4cdc-904c-37cd17e95ded" containerID="05355ffe6184f3946e526eb1fc932f2c3b6a5648acc42bb6e7aa1a059324f027" exitCode=0 Oct 30 19:00:59.557070 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:59.557005 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-743e0b-85c6498fc5-hdpjx" event={"ID":"4b1ff70c-fcdf-4cdc-904c-37cd17e95ded","Type":"ContainerDied","Data":"05355ffe6184f3946e526eb1fc932f2c3b6a5648acc42bb6e7aa1a059324f027"} Oct 30 19:00:59.758184 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:59.754324 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-743e0b-85c6498fc5-hdpjx" Oct 30 19:00:59.787006 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:59.786961 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b1ff70c-fcdf-4cdc-904c-37cd17e95ded-kube-api-access-b4ltx\" (UniqueName: \"kubernetes.io/projected/4b1ff70c-fcdf-4cdc-904c-37cd17e95ded-kube-api-access-b4ltx\") pod \"4b1ff70c-fcdf-4cdc-904c-37cd17e95ded\" (UID: \"4b1ff70c-fcdf-4cdc-904c-37cd17e95ded\") " Oct 30 19:00:59.787311 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:59.787292 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b1ff70c-fcdf-4cdc-904c-37cd17e95ded-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b1ff70c-fcdf-4cdc-904c-37cd17e95ded-plugins-config-volume\") pod \"4b1ff70c-fcdf-4cdc-904c-37cd17e95ded\" (UID: \"4b1ff70c-fcdf-4cdc-904c-37cd17e95ded\") " Oct 30 19:00:59.788429 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:59.788401 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b1ff70c-fcdf-4cdc-904c-37cd17e95ded-plugins-config-volume" pod "4b1ff70c-fcdf-4cdc-904c-37cd17e95ded" (UID: "4b1ff70c-fcdf-4cdc-904c-37cd17e95ded"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:00:59.812726 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:59.812668 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b1ff70c-fcdf-4cdc-904c-37cd17e95ded-kube-api-access-b4ltx" pod "4b1ff70c-fcdf-4cdc-904c-37cd17e95ded" (UID: "4b1ff70c-fcdf-4cdc-904c-37cd17e95ded"). InnerVolumeSpecName "kube-api-access-b4ltx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:59.825416 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:59.825396 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-743e0b/explicit-model-743e0b-kserve-78dbdb9b9c-tlgh4" Oct 30 19:00:59.889343 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:59.889309 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91470ebb-aec0-4a9f-b5d3-44a82675d02c-kube-api-access-88qp5\" (UniqueName: \"kubernetes.io/projected/91470ebb-aec0-4a9f-b5d3-44a82675d02c-kube-api-access-88qp5\") pod \"91470ebb-aec0-4a9f-b5d3-44a82675d02c\" (UID: \"91470ebb-aec0-4a9f-b5d3-44a82675d02c\") " Oct 30 19:00:59.889701 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:59.889409 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91470ebb-aec0-4a9f-b5d3-44a82675d02c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91470ebb-aec0-4a9f-b5d3-44a82675d02c-model-cache\") pod \"91470ebb-aec0-4a9f-b5d3-44a82675d02c\" (UID: \"91470ebb-aec0-4a9f-b5d3-44a82675d02c\") " Oct 30 19:00:59.889701 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:59.889675 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91470ebb-aec0-4a9f-b5d3-44a82675d02c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/91470ebb-aec0-4a9f-b5d3-44a82675d02c-dshm\") pod \"91470ebb-aec0-4a9f-b5d3-44a82675d02c\" (UID: \"91470ebb-aec0-4a9f-b5d3-44a82675d02c\") " Oct 30 19:00:59.889887 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:59.889723 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91470ebb-aec0-4a9f-b5d3-44a82675d02c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91470ebb-aec0-4a9f-b5d3-44a82675d02c-tmp-dir\") pod \"91470ebb-aec0-4a9f-b5d3-44a82675d02c\" (UID: \"91470ebb-aec0-4a9f-b5d3-44a82675d02c\") " Oct 30 19:00:59.889887 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:59.889802 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91470ebb-aec0-4a9f-b5d3-44a82675d02c-tls-certs\" (UniqueName: \"kubernetes.io/secret/91470ebb-aec0-4a9f-b5d3-44a82675d02c-tls-certs\") pod \"91470ebb-aec0-4a9f-b5d3-44a82675d02c\" (UID: \"91470ebb-aec0-4a9f-b5d3-44a82675d02c\") " Oct 30 19:00:59.889887 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:59.889831 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91470ebb-aec0-4a9f-b5d3-44a82675d02c-home\" (UniqueName: \"kubernetes.io/empty-dir/91470ebb-aec0-4a9f-b5d3-44a82675d02c-home\") pod \"91470ebb-aec0-4a9f-b5d3-44a82675d02c\" (UID: \"91470ebb-aec0-4a9f-b5d3-44a82675d02c\") " Oct 30 19:00:59.890054 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:59.889885 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91470ebb-aec0-4a9f-b5d3-44a82675d02c-model-cache" pod "91470ebb-aec0-4a9f-b5d3-44a82675d02c" (UID: "91470ebb-aec0-4a9f-b5d3-44a82675d02c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:00:59.890111 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:59.890055 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91470ebb-aec0-4a9f-b5d3-44a82675d02c-tmp-dir" pod "91470ebb-aec0-4a9f-b5d3-44a82675d02c" (UID: "91470ebb-aec0-4a9f-b5d3-44a82675d02c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:00:59.890441 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:59.890415 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4ltx\" (UniqueName: \"kubernetes.io/projected/4b1ff70c-fcdf-4cdc-904c-37cd17e95ded-kube-api-access-b4ltx\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:00:59.890524 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:59.890448 6960 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91470ebb-aec0-4a9f-b5d3-44a82675d02c-model-cache\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:00:59.890524 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:59.890464 6960 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91470ebb-aec0-4a9f-b5d3-44a82675d02c-tmp-dir\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:00:59.890524 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:59.890478 6960 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b1ff70c-fcdf-4cdc-904c-37cd17e95ded-plugins-config-volume\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:00:59.891240 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:59.891210 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91470ebb-aec0-4a9f-b5d3-44a82675d02c-home" pod "91470ebb-aec0-4a9f-b5d3-44a82675d02c" (UID: "91470ebb-aec0-4a9f-b5d3-44a82675d02c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:00:59.892721 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:59.892703 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91470ebb-aec0-4a9f-b5d3-44a82675d02c-dshm" pod "91470ebb-aec0-4a9f-b5d3-44a82675d02c" (UID: "91470ebb-aec0-4a9f-b5d3-44a82675d02c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:00:59.892885 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:59.892863 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91470ebb-aec0-4a9f-b5d3-44a82675d02c-kube-api-access-88qp5" pod "91470ebb-aec0-4a9f-b5d3-44a82675d02c" (UID: "91470ebb-aec0-4a9f-b5d3-44a82675d02c"). InnerVolumeSpecName "kube-api-access-88qp5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:59.893089 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:59.893065 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91470ebb-aec0-4a9f-b5d3-44a82675d02c-tls-certs" pod "91470ebb-aec0-4a9f-b5d3-44a82675d02c" (UID: "91470ebb-aec0-4a9f-b5d3-44a82675d02c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:00:59.992262 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:59.992180 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88qp5\" (UniqueName: \"kubernetes.io/projected/91470ebb-aec0-4a9f-b5d3-44a82675d02c-kube-api-access-88qp5\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:00:59.992262 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:59.992218 6960 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91470ebb-aec0-4a9f-b5d3-44a82675d02c-dshm\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:00:59.992262 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:59.992234 6960 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91470ebb-aec0-4a9f-b5d3-44a82675d02c-tls-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:00:59.992262 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:00:59.992246 6960 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91470ebb-aec0-4a9f-b5d3-44a82675d02c-home\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:01:00.035587 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:00.035559 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:00.565844 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:00.565814 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-743e0b-85c6498fc5-hdpjx" Oct 30 19:01:00.566011 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:00.565817 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-743e0b-85c6498fc5-hdpjx" event={"ID":"4b1ff70c-fcdf-4cdc-904c-37cd17e95ded","Type":"ContainerDied","Data":"ca257970f8026dbb8b5b3e3e08bfa44c43ba4205ff5681df5400ea4e679ef2e4"} Oct 30 19:01:00.566068 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:00.566012 6960 scope.go:122] "RemoveContainer" containerID="05355ffe6184f3946e526eb1fc932f2c3b6a5648acc42bb6e7aa1a059324f027" Oct 30 19:01:00.567444 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:00.567351 6960 generic.go:356] "Generic (PLEG): container finished" podID="91470ebb-aec0-4a9f-b5d3-44a82675d02c" containerID="398480d49483db69d4222440b23380babadad7fa5f720a214d0e7692a3b527de" exitCode=0 Oct 30 19:01:00.567444 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:00.567410 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-743e0b/explicit-model-743e0b-kserve-78dbdb9b9c-tlgh4" Oct 30 19:01:00.567554 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:00.567435 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-743e0b/explicit-model-743e0b-kserve-78dbdb9b9c-tlgh4" event={"ID":"91470ebb-aec0-4a9f-b5d3-44a82675d02c","Type":"ContainerDied","Data":"398480d49483db69d4222440b23380babadad7fa5f720a214d0e7692a3b527de"} Oct 30 19:01:00.567554 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:00.567476 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-743e0b/explicit-model-743e0b-kserve-78dbdb9b9c-tlgh4" event={"ID":"91470ebb-aec0-4a9f-b5d3-44a82675d02c","Type":"ContainerDied","Data":"a3e34159a7e26d048f2e540431bae73544e1c38a4e1fae8a9396eb232c9f894a"} Oct 30 19:01:00.577291 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:00.577274 6960 scope.go:122] "RemoveContainer" containerID="398480d49483db69d4222440b23380babadad7fa5f720a214d0e7692a3b527de" Oct 30 19:01:00.587318 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:00.587305 6960 scope.go:122] "RemoveContainer" containerID="398480d49483db69d4222440b23380babadad7fa5f720a214d0e7692a3b527de" Oct 30 19:01:00.587560 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:01:00.587541 6960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"398480d49483db69d4222440b23380babadad7fa5f720a214d0e7692a3b527de\": container with ID starting with 398480d49483db69d4222440b23380babadad7fa5f720a214d0e7692a3b527de not found: ID does not exist" containerID="398480d49483db69d4222440b23380babadad7fa5f720a214d0e7692a3b527de" Oct 30 19:01:00.587604 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:00.587570 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"398480d49483db69d4222440b23380babadad7fa5f720a214d0e7692a3b527de"} err="failed to get container status \"398480d49483db69d4222440b23380babadad7fa5f720a214d0e7692a3b527de\": rpc error: code = NotFound desc = could not find container \"398480d49483db69d4222440b23380babadad7fa5f720a214d0e7692a3b527de\": container with ID starting with 398480d49483db69d4222440b23380babadad7fa5f720a214d0e7692a3b527de not found: ID does not exist" Oct 30 19:01:00.595008 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:00.594987 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-743e0b-85c6498fc5-hdpjx"] Oct 30 19:01:00.600252 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:00.600233 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-743e0b-85c6498fc5-hdpjx"] Oct 30 19:01:00.609368 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:00.609351 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-743e0b/explicit-model-743e0b-kserve-78dbdb9b9c-tlgh4"] Oct 30 19:01:00.625280 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:00.625264 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-743e0b/explicit-model-743e0b-kserve-78dbdb9b9c-tlgh4"] Oct 30 19:01:00.952514 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:00.952479 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:01.809191 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:01.807307 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:02.594573 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:02.594546 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b1ff70c-fcdf-4cdc-904c-37cd17e95ded" path="/var/lib/kubelet/pods/4b1ff70c-fcdf-4cdc-904c-37cd17e95ded/volumes" Oct 30 19:01:02.595052 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:02.595038 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91470ebb-aec0-4a9f-b5d3-44a82675d02c" path="/var/lib/kubelet/pods/91470ebb-aec0-4a9f-b5d3-44a82675d02c/volumes" Oct 30 19:01:02.850329 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:02.850236 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:03.784094 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:03.783981 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:04.825394 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:04.825331 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:05.658169 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:05.657492 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:06.385881 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:06.385842 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-784fbf/auth-test-model-784fbf-kserve-7894bd84d9-2qwx4"] Oct 30 19:01:06.386830 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:06.386807 6960 cpu_manager.go:414] "containerMap: removing container" podUID="4b1ff70c-fcdf-4cdc-904c-37cd17e95ded" containerName="payload-pre-processing" Oct 30 19:01:06.386946 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:06.386834 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b1ff70c-fcdf-4cdc-904c-37cd17e95ded" containerName="payload-pre-processing" Oct 30 19:01:06.386946 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:06.386858 6960 cpu_manager.go:414] "containerMap: removing container" podUID="91470ebb-aec0-4a9f-b5d3-44a82675d02c" containerName="main" Oct 30 19:01:06.386946 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:06.386873 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91470ebb-aec0-4a9f-b5d3-44a82675d02c" containerName="main" Oct 30 19:01:06.387132 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:06.387116 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b1ff70c-fcdf-4cdc-904c-37cd17e95ded" containerName="payload-pre-processing" Oct 30 19:01:06.387229 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:06.387136 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="91470ebb-aec0-4a9f-b5d3-44a82675d02c" containerName="main" Oct 30 19:01:06.393040 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:06.393009 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-784fbf/auth-test-model-784fbf-kserve-7894bd84d9-2qwx4" Oct 30 19:01:06.399815 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:06.399793 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-784fbf\"/\"default-dockercfg-zhk6l\"" Oct 30 19:01:06.400061 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:06.400029 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-784fbf\"/\"kube-root-ca.crt\"" Oct 30 19:01:06.400227 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:06.400103 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-784fbf\"/\"openshift-service-ca.crt\"" Oct 30 19:01:06.403517 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:06.403492 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-784fbf/auth-test-model-784fbf-kserve-7894bd84d9-2qwx4"] Oct 30 19:01:06.403637 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:06.403575 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-784fbf\"/\"auth-test-model-784fbf-kserve-self-signed-certs\"" Oct 30 19:01:06.471828 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:06.471797 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:06.494633 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:06.494601 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fd1163d-2e1c-4889-b979-6a86786ba79c-home\") pod \"auth-test-model-784fbf-kserve-7894bd84d9-2qwx4\" (UID: \"4fd1163d-2e1c-4889-b979-6a86786ba79c\") " pod="ai-tenant-e2e-shared-b-w1-784fbf/auth-test-model-784fbf-kserve-7894bd84d9-2qwx4" Oct 30 19:01:06.494818 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:06.494693 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fd1163d-2e1c-4889-b979-6a86786ba79c-model-cache\") pod \"auth-test-model-784fbf-kserve-7894bd84d9-2qwx4\" (UID: \"4fd1163d-2e1c-4889-b979-6a86786ba79c\") " pod="ai-tenant-e2e-shared-b-w1-784fbf/auth-test-model-784fbf-kserve-7894bd84d9-2qwx4" Oct 30 19:01:06.494896 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:06.494816 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fd1163d-2e1c-4889-b979-6a86786ba79c-tmp-dir\") pod \"auth-test-model-784fbf-kserve-7894bd84d9-2qwx4\" (UID: \"4fd1163d-2e1c-4889-b979-6a86786ba79c\") " pod="ai-tenant-e2e-shared-b-w1-784fbf/auth-test-model-784fbf-kserve-7894bd84d9-2qwx4" Oct 30 19:01:06.494956 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:06.494918 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fd1163d-2e1c-4889-b979-6a86786ba79c-tls-certs\") pod \"auth-test-model-784fbf-kserve-7894bd84d9-2qwx4\" (UID: \"4fd1163d-2e1c-4889-b979-6a86786ba79c\") " pod="ai-tenant-e2e-shared-b-w1-784fbf/auth-test-model-784fbf-kserve-7894bd84d9-2qwx4" Oct 30 19:01:06.495018 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:06.494966 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fd1163d-2e1c-4889-b979-6a86786ba79c-dshm\") pod \"auth-test-model-784fbf-kserve-7894bd84d9-2qwx4\" (UID: \"4fd1163d-2e1c-4889-b979-6a86786ba79c\") " pod="ai-tenant-e2e-shared-b-w1-784fbf/auth-test-model-784fbf-kserve-7894bd84d9-2qwx4" Oct 30 19:01:06.495080 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:06.495043 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-km62k\" (UniqueName: \"kubernetes.io/projected/4fd1163d-2e1c-4889-b979-6a86786ba79c-kube-api-access-km62k\") pod \"auth-test-model-784fbf-kserve-7894bd84d9-2qwx4\" (UID: \"4fd1163d-2e1c-4889-b979-6a86786ba79c\") " pod="ai-tenant-e2e-shared-b-w1-784fbf/auth-test-model-784fbf-kserve-7894bd84d9-2qwx4" Oct 30 19:01:06.603170 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:06.598125 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fd1163d-2e1c-4889-b979-6a86786ba79c-dshm\") pod \"auth-test-model-784fbf-kserve-7894bd84d9-2qwx4\" (UID: \"4fd1163d-2e1c-4889-b979-6a86786ba79c\") " pod="ai-tenant-e2e-shared-b-w1-784fbf/auth-test-model-784fbf-kserve-7894bd84d9-2qwx4" Oct 30 19:01:06.603170 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:06.598275 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km62k\" (UniqueName: \"kubernetes.io/projected/4fd1163d-2e1c-4889-b979-6a86786ba79c-kube-api-access-km62k\") pod \"auth-test-model-784fbf-kserve-7894bd84d9-2qwx4\" (UID: \"4fd1163d-2e1c-4889-b979-6a86786ba79c\") " pod="ai-tenant-e2e-shared-b-w1-784fbf/auth-test-model-784fbf-kserve-7894bd84d9-2qwx4" Oct 30 19:01:06.603170 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:06.598392 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fd1163d-2e1c-4889-b979-6a86786ba79c-home\") pod \"auth-test-model-784fbf-kserve-7894bd84d9-2qwx4\" (UID: \"4fd1163d-2e1c-4889-b979-6a86786ba79c\") " pod="ai-tenant-e2e-shared-b-w1-784fbf/auth-test-model-784fbf-kserve-7894bd84d9-2qwx4" Oct 30 19:01:06.603170 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:06.598471 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fd1163d-2e1c-4889-b979-6a86786ba79c-model-cache\") pod \"auth-test-model-784fbf-kserve-7894bd84d9-2qwx4\" (UID: \"4fd1163d-2e1c-4889-b979-6a86786ba79c\") " pod="ai-tenant-e2e-shared-b-w1-784fbf/auth-test-model-784fbf-kserve-7894bd84d9-2qwx4" Oct 30 19:01:06.603170 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:06.598560 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fd1163d-2e1c-4889-b979-6a86786ba79c-tmp-dir\") pod \"auth-test-model-784fbf-kserve-7894bd84d9-2qwx4\" (UID: \"4fd1163d-2e1c-4889-b979-6a86786ba79c\") " pod="ai-tenant-e2e-shared-b-w1-784fbf/auth-test-model-784fbf-kserve-7894bd84d9-2qwx4" Oct 30 19:01:06.603170 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:06.598681 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fd1163d-2e1c-4889-b979-6a86786ba79c-tls-certs\") pod \"auth-test-model-784fbf-kserve-7894bd84d9-2qwx4\" (UID: \"4fd1163d-2e1c-4889-b979-6a86786ba79c\") " pod="ai-tenant-e2e-shared-b-w1-784fbf/auth-test-model-784fbf-kserve-7894bd84d9-2qwx4" Oct 30 19:01:06.603170 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:06.599576 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fd1163d-2e1c-4889-b979-6a86786ba79c-home\") pod \"auth-test-model-784fbf-kserve-7894bd84d9-2qwx4\" (UID: \"4fd1163d-2e1c-4889-b979-6a86786ba79c\") " pod="ai-tenant-e2e-shared-b-w1-784fbf/auth-test-model-784fbf-kserve-7894bd84d9-2qwx4" Oct 30 19:01:06.603170 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:06.600405 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fd1163d-2e1c-4889-b979-6a86786ba79c-model-cache\") pod \"auth-test-model-784fbf-kserve-7894bd84d9-2qwx4\" (UID: \"4fd1163d-2e1c-4889-b979-6a86786ba79c\") " pod="ai-tenant-e2e-shared-b-w1-784fbf/auth-test-model-784fbf-kserve-7894bd84d9-2qwx4" Oct 30 19:01:06.603170 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:06.600626 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fd1163d-2e1c-4889-b979-6a86786ba79c-tmp-dir\") pod \"auth-test-model-784fbf-kserve-7894bd84d9-2qwx4\" (UID: \"4fd1163d-2e1c-4889-b979-6a86786ba79c\") " pod="ai-tenant-e2e-shared-b-w1-784fbf/auth-test-model-784fbf-kserve-7894bd84d9-2qwx4" Oct 30 19:01:06.659669 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:06.640671 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fd1163d-2e1c-4889-b979-6a86786ba79c-dshm\") pod \"auth-test-model-784fbf-kserve-7894bd84d9-2qwx4\" (UID: \"4fd1163d-2e1c-4889-b979-6a86786ba79c\") " pod="ai-tenant-e2e-shared-b-w1-784fbf/auth-test-model-784fbf-kserve-7894bd84d9-2qwx4" Oct 30 19:01:06.659669 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:06.641198 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fd1163d-2e1c-4889-b979-6a86786ba79c-tls-certs\") pod \"auth-test-model-784fbf-kserve-7894bd84d9-2qwx4\" (UID: \"4fd1163d-2e1c-4889-b979-6a86786ba79c\") " pod="ai-tenant-e2e-shared-b-w1-784fbf/auth-test-model-784fbf-kserve-7894bd84d9-2qwx4" Oct 30 19:01:06.661046 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:06.661001 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-km62k\" (UniqueName: \"kubernetes.io/projected/4fd1163d-2e1c-4889-b979-6a86786ba79c-kube-api-access-km62k\") pod \"auth-test-model-784fbf-kserve-7894bd84d9-2qwx4\" (UID: \"4fd1163d-2e1c-4889-b979-6a86786ba79c\") " pod="ai-tenant-e2e-shared-b-w1-784fbf/auth-test-model-784fbf-kserve-7894bd84d9-2qwx4" Oct 30 19:01:06.716164 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:06.716080 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-784fbf/auth-test-model-784fbf-kserve-7894bd84d9-2qwx4" Oct 30 19:01:06.958546 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:06.958469 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:07.034835 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:07.034794 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-784fbf/auth-test-model-784fbf-kserve-7894bd84d9-2qwx4"] Oct 30 19:01:07.036577 ip-10-0-140-163 kubenswrapper[6960]: W1030 19:01:07.036550 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4fd1163d_2e1c_4889_b979_6a86786ba79c.slice/crio-a44d4656bc2c455d2a49c80cecdbbc82f7c74fa99781940e089e1e957e901cc0 WatchSource:0}: Error finding container a44d4656bc2c455d2a49c80cecdbbc82f7c74fa99781940e089e1e957e901cc0: Status 404 returned error can't find the container with id a44d4656bc2c455d2a49c80cecdbbc82f7c74fa99781940e089e1e957e901cc0 Oct 30 19:01:07.681191 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:07.681126 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-784fbf/auth-test-model-784fbf-kserve-7894bd84d9-2qwx4" event={"ID":"4fd1163d-2e1c-4889-b979-6a86786ba79c","Type":"ContainerStarted","Data":"eb0188677df967a7fb13138076a31bc68549995eb52a270c7ae4b33150d495b6"} Oct 30 19:01:07.681679 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:07.681198 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-784fbf/auth-test-model-784fbf-kserve-7894bd84d9-2qwx4" event={"ID":"4fd1163d-2e1c-4889-b979-6a86786ba79c","Type":"ContainerStarted","Data":"a44d4656bc2c455d2a49c80cecdbbc82f7c74fa99781940e089e1e957e901cc0"} Oct 30 19:01:07.716553 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:07.716515 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-784fbf/auth-test-model-784fbf-kserve-7894bd84d9-2qwx4" Oct 30 19:01:07.717846 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:07.717808 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-784fbf/auth-test-model-784fbf-kserve-7894bd84d9-2qwx4" podStartSLOduration=1.717795344 podStartE2EDuration="1.717795344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:01:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:01:07.714643523 +0000 UTC m=+1635.243674660" watchObservedRunningTime="2026-10-30 19:01:07.717795344 +0000 UTC m=+1635.246826482" Oct 30 19:01:07.819880 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:07.816676 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-743e0b-646fdd7b4f-5znrc"] Oct 30 19:01:07.819880 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:07.817001 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-743e0b-646fdd7b4f-5znrc" podUID="6218b805-ff40-4951-a4e2-5df7cd1e4102" containerName="maas-api" containerID="cri-o://baacb588eeede2a272f08a2b5fd946377302f5cdedd63b923aa3ed2c2fdaa6d6" gracePeriod=30 Oct 30 19:01:07.913486 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:07.913455 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:08.134224 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:08.130288 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:08.315227 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:08.311624 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:08.395676 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:08.395646 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-743e0b-646fdd7b4f-5znrc" Oct 30 19:01:08.454468 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:08.454412 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6218b805-ff40-4951-a4e2-5df7cd1e4102-kube-api-access-v4k8p\" (UniqueName: \"kubernetes.io/projected/6218b805-ff40-4951-a4e2-5df7cd1e4102-kube-api-access-v4k8p\") pod \"6218b805-ff40-4951-a4e2-5df7cd1e4102\" (UID: \"6218b805-ff40-4951-a4e2-5df7cd1e4102\") " Oct 30 19:01:08.454652 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:08.454478 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6218b805-ff40-4951-a4e2-5df7cd1e4102-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6218b805-ff40-4951-a4e2-5df7cd1e4102-openshift-service-ca\") pod \"6218b805-ff40-4951-a4e2-5df7cd1e4102\" (UID: \"6218b805-ff40-4951-a4e2-5df7cd1e4102\") " Oct 30 19:01:08.454751 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:08.454728 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6218b805-ff40-4951-a4e2-5df7cd1e4102-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6218b805-ff40-4951-a4e2-5df7cd1e4102-metrics-certs\") pod \"6218b805-ff40-4951-a4e2-5df7cd1e4102\" (UID: \"6218b805-ff40-4951-a4e2-5df7cd1e4102\") " Oct 30 19:01:08.454934 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:08.454876 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6218b805-ff40-4951-a4e2-5df7cd1e4102-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6218b805-ff40-4951-a4e2-5df7cd1e4102-maas-api-tls\") pod \"6218b805-ff40-4951-a4e2-5df7cd1e4102\" (UID: \"6218b805-ff40-4951-a4e2-5df7cd1e4102\") " Oct 30 19:01:08.455859 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:08.455829 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6218b805-ff40-4951-a4e2-5df7cd1e4102-openshift-service-ca" pod "6218b805-ff40-4951-a4e2-5df7cd1e4102" (UID: "6218b805-ff40-4951-a4e2-5df7cd1e4102"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:01:08.463607 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:08.463581 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6218b805-ff40-4951-a4e2-5df7cd1e4102-metrics-certs" pod "6218b805-ff40-4951-a4e2-5df7cd1e4102" (UID: "6218b805-ff40-4951-a4e2-5df7cd1e4102"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:01:08.464934 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:08.464910 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6218b805-ff40-4951-a4e2-5df7cd1e4102-kube-api-access-v4k8p" pod "6218b805-ff40-4951-a4e2-5df7cd1e4102" (UID: "6218b805-ff40-4951-a4e2-5df7cd1e4102"). InnerVolumeSpecName "kube-api-access-v4k8p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:01:08.465234 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:08.465215 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6218b805-ff40-4951-a4e2-5df7cd1e4102-maas-api-tls" pod "6218b805-ff40-4951-a4e2-5df7cd1e4102" (UID: "6218b805-ff40-4951-a4e2-5df7cd1e4102"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:01:08.557299 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:08.557269 6960 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6218b805-ff40-4951-a4e2-5df7cd1e4102-openshift-service-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:01:08.557299 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:08.557300 6960 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6218b805-ff40-4951-a4e2-5df7cd1e4102-metrics-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:01:08.557547 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:08.557315 6960 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6218b805-ff40-4951-a4e2-5df7cd1e4102-maas-api-tls\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:01:08.557547 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:08.557327 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4k8p\" (UniqueName: \"kubernetes.io/projected/6218b805-ff40-4951-a4e2-5df7cd1e4102-kube-api-access-v4k8p\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:01:08.640028 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:08.639991 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:08.689651 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:08.689617 6960 generic.go:356] "Generic (PLEG): container finished" podID="6218b805-ff40-4951-a4e2-5df7cd1e4102" containerID="baacb588eeede2a272f08a2b5fd946377302f5cdedd63b923aa3ed2c2fdaa6d6" exitCode=0 Oct 30 19:01:08.690109 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:08.689699 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-743e0b-646fdd7b4f-5znrc" Oct 30 19:01:08.690109 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:08.689702 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-743e0b-646fdd7b4f-5znrc" event={"ID":"6218b805-ff40-4951-a4e2-5df7cd1e4102","Type":"ContainerDied","Data":"baacb588eeede2a272f08a2b5fd946377302f5cdedd63b923aa3ed2c2fdaa6d6"} Oct 30 19:01:08.690109 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:08.689756 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-743e0b-646fdd7b4f-5znrc" event={"ID":"6218b805-ff40-4951-a4e2-5df7cd1e4102","Type":"ContainerDied","Data":"8c2313551386a1b5b57eb31429e85ae77167f1ea6672cf05285a77afcab9c93d"} Oct 30 19:01:08.690109 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:08.689778 6960 scope.go:122] "RemoveContainer" containerID="baacb588eeede2a272f08a2b5fd946377302f5cdedd63b923aa3ed2c2fdaa6d6" Oct 30 19:01:08.700667 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:08.700650 6960 scope.go:122] "RemoveContainer" containerID="baacb588eeede2a272f08a2b5fd946377302f5cdedd63b923aa3ed2c2fdaa6d6" Oct 30 19:01:08.700918 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:01:08.700900 6960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baacb588eeede2a272f08a2b5fd946377302f5cdedd63b923aa3ed2c2fdaa6d6\": container with ID starting with baacb588eeede2a272f08a2b5fd946377302f5cdedd63b923aa3ed2c2fdaa6d6 not found: ID does not exist" containerID="baacb588eeede2a272f08a2b5fd946377302f5cdedd63b923aa3ed2c2fdaa6d6" Oct 30 19:01:08.700975 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:08.700927 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baacb588eeede2a272f08a2b5fd946377302f5cdedd63b923aa3ed2c2fdaa6d6"} err="failed to get container status \"baacb588eeede2a272f08a2b5fd946377302f5cdedd63b923aa3ed2c2fdaa6d6\": rpc error: code = NotFound desc = could not find container \"baacb588eeede2a272f08a2b5fd946377302f5cdedd63b923aa3ed2c2fdaa6d6\": container with ID starting with baacb588eeede2a272f08a2b5fd946377302f5cdedd63b923aa3ed2c2fdaa6d6 not found: ID does not exist" Oct 30 19:01:08.718267 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:08.718227 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-743e0b-646fdd7b4f-5znrc"] Oct 30 19:01:08.728264 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:08.728207 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-743e0b-646fdd7b4f-5znrc"] Oct 30 19:01:09.640738 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:09.640706 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:10.558185 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:10.555033 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:10.600236 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:10.600205 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6218b805-ff40-4951-a4e2-5df7cd1e4102" path="/var/lib/kubelet/pods/6218b805-ff40-4951-a4e2-5df7cd1e4102/volumes" Oct 30 19:01:11.862943 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:11.862910 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:12.618570 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:12.618531 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:13.639932 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:13.639898 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:14.511002 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:14.510963 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:15.427773 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:15.427740 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:16.271618 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:16.271578 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:16.471861 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:16.471825 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:16.604278 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:16.604250 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-m7qgw"] Oct 30 19:01:16.604480 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:16.604446 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-m7qgw" podUID="5fcd353e-568f-4a3e-b1c6-1f44f45f5e57" containerName="istio-proxy" containerID="cri-o://4c767240e4fa49b29dd6e271eea3afab0088fbda81277e0ec5d61dcfa3b7b531" gracePeriod=30 Oct 30 19:01:16.631741 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:16.631709 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:16.723541 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:16.723512 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-784fbf/auth-test-model-784fbf-kserve-7894bd84d9-2qwx4" Oct 30 19:01:16.785259 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:16.785213 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:16.815272 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:16.815233 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-784fbf/auth-test-model-784fbf-kserve-7894bd84d9-2qwx4" Oct 30 19:01:16.856945 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:16.856856 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-784fbf/auth-test-model-784fbf-kserve-7894bd84d9-2qwx4" Oct 30 19:01:17.347639 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:17.347601 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:17.541624 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:17.541589 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:17.869103 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:17.869066 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:18.010162 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:18.010065 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:18.676679 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:18.676644 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:19.478709 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:19.478671 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:19.593333 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:19.593297 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:19.719170 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:19.718675 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:19.841203 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:19.841169 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:20.522064 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:20.522030 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:21.517051 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:21.517009 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:21.889629 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:21.889606 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-m7qgw" Oct 30 19:01:21.909516 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:21.909485 6960 generic.go:356] "Generic (PLEG): container finished" podID="5fcd353e-568f-4a3e-b1c6-1f44f45f5e57" containerID="4c767240e4fa49b29dd6e271eea3afab0088fbda81277e0ec5d61dcfa3b7b531" exitCode=0 Oct 30 19:01:21.909802 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:21.909549 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-m7qgw" event={"ID":"5fcd353e-568f-4a3e-b1c6-1f44f45f5e57","Type":"ContainerDied","Data":"4c767240e4fa49b29dd6e271eea3afab0088fbda81277e0ec5d61dcfa3b7b531"} Oct 30 19:01:21.909802 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:21.909600 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-m7qgw" event={"ID":"5fcd353e-568f-4a3e-b1c6-1f44f45f5e57","Type":"ContainerDied","Data":"b89888d28ece8688f882c4e61eeb63b8f8171751687ed062385bd30765df003e"} Oct 30 19:01:21.909802 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:21.909635 6960 scope.go:122] "RemoveContainer" containerID="4c767240e4fa49b29dd6e271eea3afab0088fbda81277e0ec5d61dcfa3b7b531" Oct 30 19:01:21.909802 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:21.909682 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-m7qgw" Oct 30 19:01:21.923692 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:21.923667 6960 scope.go:122] "RemoveContainer" containerID="4c767240e4fa49b29dd6e271eea3afab0088fbda81277e0ec5d61dcfa3b7b531" Oct 30 19:01:21.924027 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:01:21.924004 6960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c767240e4fa49b29dd6e271eea3afab0088fbda81277e0ec5d61dcfa3b7b531\": container with ID starting with 4c767240e4fa49b29dd6e271eea3afab0088fbda81277e0ec5d61dcfa3b7b531 not found: ID does not exist" containerID="4c767240e4fa49b29dd6e271eea3afab0088fbda81277e0ec5d61dcfa3b7b531" Oct 30 19:01:21.924121 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:21.924035 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c767240e4fa49b29dd6e271eea3afab0088fbda81277e0ec5d61dcfa3b7b531"} err="failed to get container status \"4c767240e4fa49b29dd6e271eea3afab0088fbda81277e0ec5d61dcfa3b7b531\": rpc error: code = NotFound desc = could not find container \"4c767240e4fa49b29dd6e271eea3afab0088fbda81277e0ec5d61dcfa3b7b531\": container with ID starting with 4c767240e4fa49b29dd6e271eea3afab0088fbda81277e0ec5d61dcfa3b7b531 not found: ID does not exist" Oct 30 19:01:22.020476 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:22.020445 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fcd353e-568f-4a3e-b1c6-1f44f45f5e57-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fcd353e-568f-4a3e-b1c6-1f44f45f5e57-credential-socket\") pod \"5fcd353e-568f-4a3e-b1c6-1f44f45f5e57\" (UID: \"5fcd353e-568f-4a3e-b1c6-1f44f45f5e57\") " Oct 30 19:01:22.020663 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:22.020515 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fcd353e-568f-4a3e-b1c6-1f44f45f5e57-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fcd353e-568f-4a3e-b1c6-1f44f45f5e57-istiod-ca-cert\") pod \"5fcd353e-568f-4a3e-b1c6-1f44f45f5e57\" (UID: \"5fcd353e-568f-4a3e-b1c6-1f44f45f5e57\") " Oct 30 19:01:22.020741 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:22.020681 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fcd353e-568f-4a3e-b1c6-1f44f45f5e57-kube-api-access-sbxzw\" (UniqueName: \"kubernetes.io/projected/5fcd353e-568f-4a3e-b1c6-1f44f45f5e57-kube-api-access-sbxzw\") pod \"5fcd353e-568f-4a3e-b1c6-1f44f45f5e57\" (UID: \"5fcd353e-568f-4a3e-b1c6-1f44f45f5e57\") " Oct 30 19:01:22.020794 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:22.020744 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fcd353e-568f-4a3e-b1c6-1f44f45f5e57-istio-token\" (UniqueName: \"kubernetes.io/projected/5fcd353e-568f-4a3e-b1c6-1f44f45f5e57-istio-token\") pod \"5fcd353e-568f-4a3e-b1c6-1f44f45f5e57\" (UID: \"5fcd353e-568f-4a3e-b1c6-1f44f45f5e57\") " Oct 30 19:01:22.020794 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:22.020760 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fcd353e-568f-4a3e-b1c6-1f44f45f5e57-credential-socket" pod "5fcd353e-568f-4a3e-b1c6-1f44f45f5e57" (UID: "5fcd353e-568f-4a3e-b1c6-1f44f45f5e57"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:01:22.020895 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:22.020815 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5fcd353e-568f-4a3e-b1c6-1f44f45f5e57-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fcd353e-568f-4a3e-b1c6-1f44f45f5e57-istio-podinfo\") pod \"5fcd353e-568f-4a3e-b1c6-1f44f45f5e57\" (UID: \"5fcd353e-568f-4a3e-b1c6-1f44f45f5e57\") " Oct 30 19:01:22.020895 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:22.020866 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fcd353e-568f-4a3e-b1c6-1f44f45f5e57-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fcd353e-568f-4a3e-b1c6-1f44f45f5e57-workload-socket\") pod \"5fcd353e-568f-4a3e-b1c6-1f44f45f5e57\" (UID: \"5fcd353e-568f-4a3e-b1c6-1f44f45f5e57\") " Oct 30 19:01:22.020996 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:22.020949 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fcd353e-568f-4a3e-b1c6-1f44f45f5e57-istiod-ca-cert" pod "5fcd353e-568f-4a3e-b1c6-1f44f45f5e57" (UID: "5fcd353e-568f-4a3e-b1c6-1f44f45f5e57"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:01:22.020996 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:22.020983 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fcd353e-568f-4a3e-b1c6-1f44f45f5e57-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fcd353e-568f-4a3e-b1c6-1f44f45f5e57-istio-envoy\") pod \"5fcd353e-568f-4a3e-b1c6-1f44f45f5e57\" (UID: \"5fcd353e-568f-4a3e-b1c6-1f44f45f5e57\") " Oct 30 19:01:22.021104 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:22.021057 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fcd353e-568f-4a3e-b1c6-1f44f45f5e57-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fcd353e-568f-4a3e-b1c6-1f44f45f5e57-istio-data\") pod \"5fcd353e-568f-4a3e-b1c6-1f44f45f5e57\" (UID: \"5fcd353e-568f-4a3e-b1c6-1f44f45f5e57\") " Oct 30 19:01:22.021104 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:22.021094 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fcd353e-568f-4a3e-b1c6-1f44f45f5e57-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fcd353e-568f-4a3e-b1c6-1f44f45f5e57-workload-certs\") pod \"5fcd353e-568f-4a3e-b1c6-1f44f45f5e57\" (UID: \"5fcd353e-568f-4a3e-b1c6-1f44f45f5e57\") " Oct 30 19:01:22.029207 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:22.025647 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fcd353e-568f-4a3e-b1c6-1f44f45f5e57-istio-data" pod "5fcd353e-568f-4a3e-b1c6-1f44f45f5e57" (UID: "5fcd353e-568f-4a3e-b1c6-1f44f45f5e57"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:01:22.032991 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:22.032963 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fcd353e-568f-4a3e-b1c6-1f44f45f5e57-kube-api-access-sbxzw" pod "5fcd353e-568f-4a3e-b1c6-1f44f45f5e57" (UID: "5fcd353e-568f-4a3e-b1c6-1f44f45f5e57"). InnerVolumeSpecName "kube-api-access-sbxzw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:01:22.038401 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:22.038356 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fcd353e-568f-4a3e-b1c6-1f44f45f5e57-workload-certs" pod "5fcd353e-568f-4a3e-b1c6-1f44f45f5e57" (UID: "5fcd353e-568f-4a3e-b1c6-1f44f45f5e57"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:01:22.038401 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:22.038364 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fcd353e-568f-4a3e-b1c6-1f44f45f5e57-workload-socket" pod "5fcd353e-568f-4a3e-b1c6-1f44f45f5e57" (UID: "5fcd353e-568f-4a3e-b1c6-1f44f45f5e57"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:01:22.042508 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:22.038875 6960 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fcd353e-568f-4a3e-b1c6-1f44f45f5e57-istio-data\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:01:22.042508 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:22.038903 6960 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fcd353e-568f-4a3e-b1c6-1f44f45f5e57-workload-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:01:22.042508 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:22.038908 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fcd353e-568f-4a3e-b1c6-1f44f45f5e57-istio-envoy" pod "5fcd353e-568f-4a3e-b1c6-1f44f45f5e57" (UID: "5fcd353e-568f-4a3e-b1c6-1f44f45f5e57"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:01:22.042508 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:22.038920 6960 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fcd353e-568f-4a3e-b1c6-1f44f45f5e57-credential-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:01:22.042508 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:22.038942 6960 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fcd353e-568f-4a3e-b1c6-1f44f45f5e57-istiod-ca-cert\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:01:22.042508 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:22.038967 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbxzw\" (UniqueName: \"kubernetes.io/projected/5fcd353e-568f-4a3e-b1c6-1f44f45f5e57-kube-api-access-sbxzw\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:01:22.042508 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:22.038982 6960 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fcd353e-568f-4a3e-b1c6-1f44f45f5e57-workload-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:01:22.051287 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:22.051202 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5fcd353e-568f-4a3e-b1c6-1f44f45f5e57-istio-podinfo" pod "5fcd353e-568f-4a3e-b1c6-1f44f45f5e57" (UID: "5fcd353e-568f-4a3e-b1c6-1f44f45f5e57"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:01:22.051287 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:22.051228 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fcd353e-568f-4a3e-b1c6-1f44f45f5e57-istio-token" pod "5fcd353e-568f-4a3e-b1c6-1f44f45f5e57" (UID: "5fcd353e-568f-4a3e-b1c6-1f44f45f5e57"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:01:22.111309 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:22.111279 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-aec752/noten-model-aec752-kserve-5f6d5744d5-c8tvl"] Oct 30 19:01:22.112182 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:22.112121 6960 cpu_manager.go:414] "containerMap: removing container" podUID="6218b805-ff40-4951-a4e2-5df7cd1e4102" containerName="maas-api" Oct 30 19:01:22.112182 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:22.112163 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6218b805-ff40-4951-a4e2-5df7cd1e4102" containerName="maas-api" Oct 30 19:01:22.112387 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:22.112197 6960 cpu_manager.go:414] "containerMap: removing container" podUID="5fcd353e-568f-4a3e-b1c6-1f44f45f5e57" containerName="istio-proxy" Oct 30 19:01:22.112387 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:22.112202 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fcd353e-568f-4a3e-b1c6-1f44f45f5e57" containerName="istio-proxy" Oct 30 19:01:22.112387 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:22.112339 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fcd353e-568f-4a3e-b1c6-1f44f45f5e57" containerName="istio-proxy" Oct 30 19:01:22.112387 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:22.112358 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="6218b805-ff40-4951-a4e2-5df7cd1e4102" containerName="maas-api" Oct 30 19:01:22.117254 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:22.117232 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-aec752/noten-model-aec752-kserve-5f6d5744d5-c8tvl" Oct 30 19:01:22.127083 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:22.127059 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-aec752\"/\"openshift-service-ca.crt\"" Oct 30 19:01:22.129384 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:22.129361 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-aec752\"/\"kube-root-ca.crt\"" Oct 30 19:01:22.131568 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:22.131536 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-aec752\"/\"noten-model-aec752-kserve-self-signed-certs\"" Oct 30 19:01:22.131751 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:22.131739 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-aec752\"/\"default-dockercfg-rfxf7\"" Oct 30 19:01:22.133042 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:22.133019 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-aec752/noten-model-aec752-kserve-5f6d5744d5-c8tvl"] Oct 30 19:01:22.140450 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:22.140413 6960 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fcd353e-568f-4a3e-b1c6-1f44f45f5e57-istio-envoy\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:01:22.140450 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:22.140440 6960 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fcd353e-568f-4a3e-b1c6-1f44f45f5e57-istio-token\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:01:22.140636 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:22.140456 6960 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fcd353e-568f-4a3e-b1c6-1f44f45f5e57-istio-podinfo\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:01:22.186900 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:22.186854 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:22.239957 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:22.239927 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-m7qgw"] Oct 30 19:01:22.241713 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:22.241683 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ftvj\" (UniqueName: \"kubernetes.io/projected/63d5aa61-3753-41d2-8538-f996d788584e-kube-api-access-7ftvj\") pod \"noten-model-aec752-kserve-5f6d5744d5-c8tvl\" (UID: \"63d5aa61-3753-41d2-8538-f996d788584e\") " pod="ai-tenant-e2e-noten-aec752/noten-model-aec752-kserve-5f6d5744d5-c8tvl" Oct 30 19:01:22.241840 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:22.241722 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63d5aa61-3753-41d2-8538-f996d788584e-model-cache\") pod \"noten-model-aec752-kserve-5f6d5744d5-c8tvl\" (UID: \"63d5aa61-3753-41d2-8538-f996d788584e\") " pod="ai-tenant-e2e-noten-aec752/noten-model-aec752-kserve-5f6d5744d5-c8tvl" Oct 30 19:01:22.241840 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:22.241808 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63d5aa61-3753-41d2-8538-f996d788584e-tmp-dir\") pod \"noten-model-aec752-kserve-5f6d5744d5-c8tvl\" (UID: \"63d5aa61-3753-41d2-8538-f996d788584e\") " pod="ai-tenant-e2e-noten-aec752/noten-model-aec752-kserve-5f6d5744d5-c8tvl" Oct 30 19:01:22.242061 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:22.242040 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63d5aa61-3753-41d2-8538-f996d788584e-dshm\") pod \"noten-model-aec752-kserve-5f6d5744d5-c8tvl\" (UID: \"63d5aa61-3753-41d2-8538-f996d788584e\") " pod="ai-tenant-e2e-noten-aec752/noten-model-aec752-kserve-5f6d5744d5-c8tvl" Oct 30 19:01:22.242134 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:22.242094 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63d5aa61-3753-41d2-8538-f996d788584e-tls-certs\") pod \"noten-model-aec752-kserve-5f6d5744d5-c8tvl\" (UID: \"63d5aa61-3753-41d2-8538-f996d788584e\") " pod="ai-tenant-e2e-noten-aec752/noten-model-aec752-kserve-5f6d5744d5-c8tvl" Oct 30 19:01:22.242134 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:22.242122 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63d5aa61-3753-41d2-8538-f996d788584e-home\") pod \"noten-model-aec752-kserve-5f6d5744d5-c8tvl\" (UID: \"63d5aa61-3753-41d2-8538-f996d788584e\") " pod="ai-tenant-e2e-noten-aec752/noten-model-aec752-kserve-5f6d5744d5-c8tvl" Oct 30 19:01:22.249013 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:22.248983 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-743e0b-openshift-default-65cdbd6b45-m7qgw"] Oct 30 19:01:22.344609 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:22.344571 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63d5aa61-3753-41d2-8538-f996d788584e-dshm\") pod \"noten-model-aec752-kserve-5f6d5744d5-c8tvl\" (UID: \"63d5aa61-3753-41d2-8538-f996d788584e\") " pod="ai-tenant-e2e-noten-aec752/noten-model-aec752-kserve-5f6d5744d5-c8tvl" Oct 30 19:01:22.344897 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:22.344878 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63d5aa61-3753-41d2-8538-f996d788584e-tls-certs\") pod \"noten-model-aec752-kserve-5f6d5744d5-c8tvl\" (UID: \"63d5aa61-3753-41d2-8538-f996d788584e\") " pod="ai-tenant-e2e-noten-aec752/noten-model-aec752-kserve-5f6d5744d5-c8tvl" Oct 30 19:01:22.345011 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:22.344997 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63d5aa61-3753-41d2-8538-f996d788584e-home\") pod \"noten-model-aec752-kserve-5f6d5744d5-c8tvl\" (UID: \"63d5aa61-3753-41d2-8538-f996d788584e\") " pod="ai-tenant-e2e-noten-aec752/noten-model-aec752-kserve-5f6d5744d5-c8tvl" Oct 30 19:01:22.345260 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:22.345235 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ftvj\" (UniqueName: \"kubernetes.io/projected/63d5aa61-3753-41d2-8538-f996d788584e-kube-api-access-7ftvj\") pod \"noten-model-aec752-kserve-5f6d5744d5-c8tvl\" (UID: \"63d5aa61-3753-41d2-8538-f996d788584e\") " pod="ai-tenant-e2e-noten-aec752/noten-model-aec752-kserve-5f6d5744d5-c8tvl" Oct 30 19:01:22.345402 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:22.345388 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63d5aa61-3753-41d2-8538-f996d788584e-model-cache\") pod \"noten-model-aec752-kserve-5f6d5744d5-c8tvl\" (UID: \"63d5aa61-3753-41d2-8538-f996d788584e\") " pod="ai-tenant-e2e-noten-aec752/noten-model-aec752-kserve-5f6d5744d5-c8tvl" Oct 30 19:01:22.345529 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:22.345516 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63d5aa61-3753-41d2-8538-f996d788584e-tmp-dir\") pod \"noten-model-aec752-kserve-5f6d5744d5-c8tvl\" (UID: \"63d5aa61-3753-41d2-8538-f996d788584e\") " pod="ai-tenant-e2e-noten-aec752/noten-model-aec752-kserve-5f6d5744d5-c8tvl" Oct 30 19:01:22.346617 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:22.346024 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63d5aa61-3753-41d2-8538-f996d788584e-home\") pod \"noten-model-aec752-kserve-5f6d5744d5-c8tvl\" (UID: \"63d5aa61-3753-41d2-8538-f996d788584e\") " pod="ai-tenant-e2e-noten-aec752/noten-model-aec752-kserve-5f6d5744d5-c8tvl" Oct 30 19:01:22.362128 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:22.347071 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63d5aa61-3753-41d2-8538-f996d788584e-model-cache\") pod \"noten-model-aec752-kserve-5f6d5744d5-c8tvl\" (UID: \"63d5aa61-3753-41d2-8538-f996d788584e\") " pod="ai-tenant-e2e-noten-aec752/noten-model-aec752-kserve-5f6d5744d5-c8tvl" Oct 30 19:01:22.362362 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:22.347385 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63d5aa61-3753-41d2-8538-f996d788584e-tmp-dir\") pod \"noten-model-aec752-kserve-5f6d5744d5-c8tvl\" (UID: \"63d5aa61-3753-41d2-8538-f996d788584e\") " pod="ai-tenant-e2e-noten-aec752/noten-model-aec752-kserve-5f6d5744d5-c8tvl" Oct 30 19:01:22.362483 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:22.361680 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ftvj\" (UniqueName: \"kubernetes.io/projected/63d5aa61-3753-41d2-8538-f996d788584e-kube-api-access-7ftvj\") pod \"noten-model-aec752-kserve-5f6d5744d5-c8tvl\" (UID: \"63d5aa61-3753-41d2-8538-f996d788584e\") " pod="ai-tenant-e2e-noten-aec752/noten-model-aec752-kserve-5f6d5744d5-c8tvl" Oct 30 19:01:22.362574 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:22.361702 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63d5aa61-3753-41d2-8538-f996d788584e-tls-certs\") pod \"noten-model-aec752-kserve-5f6d5744d5-c8tvl\" (UID: \"63d5aa61-3753-41d2-8538-f996d788584e\") " pod="ai-tenant-e2e-noten-aec752/noten-model-aec752-kserve-5f6d5744d5-c8tvl" Oct 30 19:01:22.362902 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:22.362882 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63d5aa61-3753-41d2-8538-f996d788584e-dshm\") pod \"noten-model-aec752-kserve-5f6d5744d5-c8tvl\" (UID: \"63d5aa61-3753-41d2-8538-f996d788584e\") " pod="ai-tenant-e2e-noten-aec752/noten-model-aec752-kserve-5f6d5744d5-c8tvl" Oct 30 19:01:22.439466 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:22.439429 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-aec752/noten-model-aec752-kserve-5f6d5744d5-c8tvl" Oct 30 19:01:22.464098 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:22.464055 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:22.640926 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:22.640842 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fcd353e-568f-4a3e-b1c6-1f44f45f5e57" path="/var/lib/kubelet/pods/5fcd353e-568f-4a3e-b1c6-1f44f45f5e57/volumes" Oct 30 19:01:22.755653 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:22.755620 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-aec752/noten-model-aec752-kserve-5f6d5744d5-c8tvl"] Oct 30 19:01:22.756082 ip-10-0-140-163 kubenswrapper[6960]: W1030 19:01:22.756059 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod63d5aa61_3753_41d2_8538_f996d788584e.slice/crio-a288e09ec47f3eda7e2427fd9db04084878c4cd6a5a8bbb6e7a79b8b30a85186 WatchSource:0}: Error finding container a288e09ec47f3eda7e2427fd9db04084878c4cd6a5a8bbb6e7a79b8b30a85186: Status 404 returned error can't find the container with id a288e09ec47f3eda7e2427fd9db04084878c4cd6a5a8bbb6e7a79b8b30a85186 Oct 30 19:01:22.919282 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:22.919206 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-aec752/noten-model-aec752-kserve-5f6d5744d5-c8tvl" event={"ID":"63d5aa61-3753-41d2-8538-f996d788584e","Type":"ContainerStarted","Data":"84b520becfbb4bce228700fd018a2498df41c3556fd27d9eefe954ee830649e7"} Oct 30 19:01:22.919282 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:22.919243 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-aec752/noten-model-aec752-kserve-5f6d5744d5-c8tvl" event={"ID":"63d5aa61-3753-41d2-8538-f996d788584e","Type":"ContainerStarted","Data":"a288e09ec47f3eda7e2427fd9db04084878c4cd6a5a8bbb6e7a79b8b30a85186"} Oct 30 19:01:22.965454 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:22.965406 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-noten-aec752/noten-model-aec752-kserve-5f6d5744d5-c8tvl" podStartSLOduration=0.965388816 podStartE2EDuration="965.388816ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:01:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:01:22.964494319 +0000 UTC m=+1650.493525457" watchObservedRunningTime="2026-10-30 19:01:22.965388816 +0000 UTC m=+1650.494419956" Oct 30 19:01:23.439773 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:23.439743 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-aec752/noten-model-aec752-kserve-5f6d5744d5-c8tvl" Oct 30 19:01:23.442758 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:23.442729 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:23.936163 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:01:23.936114 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 19:01:24.316022 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:24.315988 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:25.113896 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:25.113866 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:25.915021 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:25.914984 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:26.793344 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:26.793312 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:27.690409 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:27.690370 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:28.717445 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:28.717412 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:28.937404 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:28.937373 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:29.742337 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:29.742298 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:30.707268 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:30.707239 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:31.709340 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:31.709293 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:32.440268 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:32.440227 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-aec752/noten-model-aec752-kserve-5f6d5744d5-c8tvl" Oct 30 19:01:32.455713 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:32.455683 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-aec752/noten-model-aec752-kserve-5f6d5744d5-c8tvl" Oct 30 19:01:32.542211 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:32.542175 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:33.011724 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:33.011696 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-aec752/noten-model-aec752-kserve-5f6d5744d5-c8tvl" Oct 30 19:01:33.422178 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:33.422128 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:33.722526 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:33.722442 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-aec752/noten-model-aec752-kserve-5f6d5744d5-c8tvl"] Oct 30 19:01:33.777354 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:33.777289 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:34.007253 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:33.998641 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:34.128764 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:01:34.128732 6960 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-aec752/noten-model-aec752-kserve-self-signed-certs: secret "noten-model-aec752-kserve-self-signed-certs" not found Oct 30 19:01:34.129510 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:01:34.129488 6960 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63d5aa61-3753-41d2-8538-f996d788584e-tls-certs podName:63d5aa61-3753-41d2-8538-f996d788584e nodeName:}" failed. No retries permitted until 2026-10-30 19:01:34.629454924 +0000 UTC m=+1662.158486118 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/63d5aa61-3753-41d2-8538-f996d788584e-tls-certs") pod "noten-model-aec752-kserve-5f6d5744d5-c8tvl" (UID: "63d5aa61-3753-41d2-8538-f996d788584e") : secret "noten-model-aec752-kserve-self-signed-certs" not found Oct 30 19:01:34.304966 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:34.304936 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:34.456870 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:34.456839 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:34.637497 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:01:34.637431 6960 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-aec752/noten-model-aec752-kserve-self-signed-certs: secret "noten-model-aec752-kserve-self-signed-certs" not found Oct 30 19:01:34.637497 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:01:34.637489 6960 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63d5aa61-3753-41d2-8538-f996d788584e-tls-certs podName:63d5aa61-3753-41d2-8538-f996d788584e nodeName:}" failed. No retries permitted until 2026-10-30 19:01:35.637477262 +0000 UTC m=+1663.166508391 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/63d5aa61-3753-41d2-8538-f996d788584e-tls-certs") pod "noten-model-aec752-kserve-5f6d5744d5-c8tvl" (UID: "63d5aa61-3753-41d2-8538-f996d788584e") : secret "noten-model-aec752-kserve-self-signed-certs" not found Oct 30 19:01:34.642733 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:34.642695 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:35.415611 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:35.415569 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:35.652801 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:01:35.652769 6960 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-aec752/noten-model-aec752-kserve-self-signed-certs: secret "noten-model-aec752-kserve-self-signed-certs" not found Oct 30 19:01:35.652991 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:01:35.652836 6960 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63d5aa61-3753-41d2-8538-f996d788584e-tls-certs podName:63d5aa61-3753-41d2-8538-f996d788584e nodeName:}" failed. No retries permitted until 2026-10-30 19:01:37.652822258 +0000 UTC m=+1665.181853387 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/63d5aa61-3753-41d2-8538-f996d788584e-tls-certs") pod "noten-model-aec752-kserve-5f6d5744d5-c8tvl" (UID: "63d5aa61-3753-41d2-8538-f996d788584e") : secret "noten-model-aec752-kserve-self-signed-certs" not found Oct 30 19:01:36.313283 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:36.313249 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:37.135653 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:37.135613 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:37.686376 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:01:37.686335 6960 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-aec752/noten-model-aec752-kserve-self-signed-certs: secret "noten-model-aec752-kserve-self-signed-certs" not found Oct 30 19:01:37.686550 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:01:37.686419 6960 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63d5aa61-3753-41d2-8538-f996d788584e-tls-certs podName:63d5aa61-3753-41d2-8538-f996d788584e nodeName:}" failed. No retries permitted until 2026-10-30 19:01:41.686401892 +0000 UTC m=+1669.215433008 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/63d5aa61-3753-41d2-8538-f996d788584e-tls-certs") pod "noten-model-aec752-kserve-5f6d5744d5-c8tvl" (UID: "63d5aa61-3753-41d2-8538-f996d788584e") : secret "noten-model-aec752-kserve-self-signed-certs" not found Oct 30 19:01:38.052003 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:38.051925 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:39.044179 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:39.043866 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:39.532844 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:39.532811 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:40.304525 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:40.304494 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:41.312777 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:41.312741 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:41.769551 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:01:41.769474 6960 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-aec752/noten-model-aec752-kserve-self-signed-certs: secret "noten-model-aec752-kserve-self-signed-certs" not found Oct 30 19:01:41.769697 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:01:41.769612 6960 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63d5aa61-3753-41d2-8538-f996d788584e-tls-certs podName:63d5aa61-3753-41d2-8538-f996d788584e nodeName:}" failed. No retries permitted until 2026-10-30 19:01:49.769587571 +0000 UTC m=+1677.298618694 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/63d5aa61-3753-41d2-8538-f996d788584e-tls-certs") pod "noten-model-aec752-kserve-5f6d5744d5-c8tvl" (UID: "63d5aa61-3753-41d2-8538-f996d788584e") : secret "noten-model-aec752-kserve-self-signed-certs" not found Oct 30 19:01:42.214039 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:42.214004 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:43.039543 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:43.039509 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:44.008781 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:44.008748 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:44.865565 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:44.865533 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:45.831138 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:45.831091 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:46.698089 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:46.698038 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:47.545924 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:47.545890 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:48.462643 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:48.462602 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:49.233327 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:49.233295 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:49.789594 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:01:49.789561 6960 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-aec752/noten-model-aec752-kserve-self-signed-certs: secret "noten-model-aec752-kserve-self-signed-certs" not found Oct 30 19:01:49.790021 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:01:49.789643 6960 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63d5aa61-3753-41d2-8538-f996d788584e-tls-certs podName:63d5aa61-3753-41d2-8538-f996d788584e nodeName:}" failed. No retries permitted until 2026-10-30 19:02:05.789625054 +0000 UTC m=+1693.318656215 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/63d5aa61-3753-41d2-8538-f996d788584e-tls-certs") pod "noten-model-aec752-kserve-5f6d5744d5-c8tvl" (UID: "63d5aa61-3753-41d2-8538-f996d788584e") : secret "noten-model-aec752-kserve-self-signed-certs" not found Oct 30 19:01:50.011806 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:50.011765 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:50.065364 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:50.065320 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-noten-aec752/noten-model-aec752-kserve-5f6d5744d5-c8tvl" podUID="63d5aa61-3753-41d2-8538-f996d788584e" containerName="main" containerID="cri-o://84b520becfbb4bce228700fd018a2498df41c3556fd27d9eefe954ee830649e7" gracePeriod=45 Oct 30 19:01:50.191064 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:50.191030 6960 generic.go:356] "Generic (PLEG): container finished" podID="63d5aa61-3753-41d2-8538-f996d788584e" containerID="84b520becfbb4bce228700fd018a2498df41c3556fd27d9eefe954ee830649e7" exitCode=0 Oct 30 19:01:50.191257 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:50.191099 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-aec752/noten-model-aec752-kserve-5f6d5744d5-c8tvl" event={"ID":"63d5aa61-3753-41d2-8538-f996d788584e","Type":"ContainerDied","Data":"84b520becfbb4bce228700fd018a2498df41c3556fd27d9eefe954ee830649e7"} Oct 30 19:01:50.339778 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:50.339749 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-aec752/noten-model-aec752-kserve-5f6d5744d5-c8tvl" Oct 30 19:01:50.398805 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:50.398771 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63d5aa61-3753-41d2-8538-f996d788584e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63d5aa61-3753-41d2-8538-f996d788584e-model-cache\") pod \"63d5aa61-3753-41d2-8538-f996d788584e\" (UID: \"63d5aa61-3753-41d2-8538-f996d788584e\") " Oct 30 19:01:50.398981 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:50.398824 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63d5aa61-3753-41d2-8538-f996d788584e-home\" (UniqueName: \"kubernetes.io/empty-dir/63d5aa61-3753-41d2-8538-f996d788584e-home\") pod \"63d5aa61-3753-41d2-8538-f996d788584e\" (UID: \"63d5aa61-3753-41d2-8538-f996d788584e\") " Oct 30 19:01:50.399061 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:50.399005 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63d5aa61-3753-41d2-8538-f996d788584e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/63d5aa61-3753-41d2-8538-f996d788584e-dshm\") pod \"63d5aa61-3753-41d2-8538-f996d788584e\" (UID: \"63d5aa61-3753-41d2-8538-f996d788584e\") " Oct 30 19:01:50.399061 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:50.399039 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63d5aa61-3753-41d2-8538-f996d788584e-kube-api-access-7ftvj\" (UniqueName: \"kubernetes.io/projected/63d5aa61-3753-41d2-8538-f996d788584e-kube-api-access-7ftvj\") pod \"63d5aa61-3753-41d2-8538-f996d788584e\" (UID: \"63d5aa61-3753-41d2-8538-f996d788584e\") " Oct 30 19:01:50.399186 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:50.399063 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63d5aa61-3753-41d2-8538-f996d788584e-model-cache" pod "63d5aa61-3753-41d2-8538-f996d788584e" (UID: "63d5aa61-3753-41d2-8538-f996d788584e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:01:50.399259 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:50.399216 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63d5aa61-3753-41d2-8538-f996d788584e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63d5aa61-3753-41d2-8538-f996d788584e-tmp-dir\") pod \"63d5aa61-3753-41d2-8538-f996d788584e\" (UID: \"63d5aa61-3753-41d2-8538-f996d788584e\") " Oct 30 19:01:50.399259 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:50.399235 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63d5aa61-3753-41d2-8538-f996d788584e-home" pod "63d5aa61-3753-41d2-8538-f996d788584e" (UID: "63d5aa61-3753-41d2-8538-f996d788584e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:01:50.399409 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:50.399384 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63d5aa61-3753-41d2-8538-f996d788584e-tls-certs\" (UniqueName: \"kubernetes.io/secret/63d5aa61-3753-41d2-8538-f996d788584e-tls-certs\") pod \"63d5aa61-3753-41d2-8538-f996d788584e\" (UID: \"63d5aa61-3753-41d2-8538-f996d788584e\") " Oct 30 19:01:50.399577 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:50.399541 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63d5aa61-3753-41d2-8538-f996d788584e-tmp-dir" pod "63d5aa61-3753-41d2-8538-f996d788584e" (UID: "63d5aa61-3753-41d2-8538-f996d788584e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:01:50.400292 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:50.400263 6960 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63d5aa61-3753-41d2-8538-f996d788584e-model-cache\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:01:50.400292 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:50.400294 6960 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63d5aa61-3753-41d2-8538-f996d788584e-home\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:01:50.400497 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:50.400310 6960 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63d5aa61-3753-41d2-8538-f996d788584e-tmp-dir\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:01:50.401994 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:50.401970 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63d5aa61-3753-41d2-8538-f996d788584e-dshm" pod "63d5aa61-3753-41d2-8538-f996d788584e" (UID: "63d5aa61-3753-41d2-8538-f996d788584e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:01:50.402280 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:50.402256 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63d5aa61-3753-41d2-8538-f996d788584e-kube-api-access-7ftvj" pod "63d5aa61-3753-41d2-8538-f996d788584e" (UID: "63d5aa61-3753-41d2-8538-f996d788584e"). InnerVolumeSpecName "kube-api-access-7ftvj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:01:50.402368 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:50.402331 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63d5aa61-3753-41d2-8538-f996d788584e-tls-certs" pod "63d5aa61-3753-41d2-8538-f996d788584e" (UID: "63d5aa61-3753-41d2-8538-f996d788584e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:01:50.501671 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:50.501641 6960 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63d5aa61-3753-41d2-8538-f996d788584e-dshm\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:01:50.501671 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:50.501668 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ftvj\" (UniqueName: \"kubernetes.io/projected/63d5aa61-3753-41d2-8538-f996d788584e-kube-api-access-7ftvj\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:01:50.501671 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:50.501677 6960 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63d5aa61-3753-41d2-8538-f996d788584e-tls-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:01:50.825923 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:50.825891 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:51.209868 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:51.209818 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-aec752/noten-model-aec752-kserve-5f6d5744d5-c8tvl" Oct 30 19:01:51.210092 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:51.209884 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-aec752/noten-model-aec752-kserve-5f6d5744d5-c8tvl" event={"ID":"63d5aa61-3753-41d2-8538-f996d788584e","Type":"ContainerDied","Data":"a288e09ec47f3eda7e2427fd9db04084878c4cd6a5a8bbb6e7a79b8b30a85186"} Oct 30 19:01:51.210092 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:51.209938 6960 scope.go:122] "RemoveContainer" containerID="84b520becfbb4bce228700fd018a2498df41c3556fd27d9eefe954ee830649e7" Oct 30 19:01:51.233579 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:51.233557 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-aec752/noten-model-aec752-kserve-5f6d5744d5-c8tvl"] Oct 30 19:01:51.238046 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:51.238023 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-aec752/noten-model-aec752-kserve-5f6d5744d5-c8tvl"] Oct 30 19:01:51.652221 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:51.652175 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:52.395901 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:52.395855 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:52.595547 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:52.595482 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63d5aa61-3753-41d2-8538-f996d788584e" path="/var/lib/kubelet/pods/63d5aa61-3753-41d2-8538-f996d788584e/volumes" Oct 30 19:01:52.638876 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:52.638834 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:53.642431 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:53.642393 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:53.933291 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:01:53.933210 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 19:01:54.412926 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:54.412899 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:55.343919 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:55.343883 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:56.128311 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:56.128282 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:56.983885 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:56.983847 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:57.752396 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:57.752365 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:58.520098 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:58.520048 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:59.311311 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:01:59.311278 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:00.118699 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:00.118662 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:00.912771 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:00.912730 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:01.805510 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:01.805476 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:02.581369 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:02.581337 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:03.417999 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:03.417965 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:04.210789 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:04.210741 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:05.035363 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:05.035324 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:05.842397 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:05.842366 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:06.710922 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:06.710887 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:07.675420 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:07.675383 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:08.525559 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:08.525525 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:09.329524 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:09.329482 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:10.222296 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:10.222251 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:11.208797 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:11.208755 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:12.082531 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:12.082496 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:12.857382 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:12.857347 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:13.635680 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:13.635627 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:14.596625 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:14.596595 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:15.420136 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:15.420081 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:16.311050 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:16.311011 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:17.313696 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:17.313616 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:17.561699 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:17.561664 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:17.764182 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:17.764096 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7414-f00c55"] Oct 30 19:02:17.764761 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:17.764748 6960 cpu_manager.go:414] "containerMap: removing container" podUID="63d5aa61-3753-41d2-8538-f996d788584e" containerName="main" Oct 30 19:02:17.764809 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:17.764765 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63d5aa61-3753-41d2-8538-f996d788584e" containerName="main" Oct 30 19:02:17.764918 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:17.764909 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="63d5aa61-3753-41d2-8538-f996d788584e" containerName="main" Oct 30 19:02:17.768496 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:17.768476 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7414-f00c55" Oct 30 19:02:17.774288 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:17.774267 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-7h66b\"" Oct 30 19:02:17.775404 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:17.775383 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7414-f00c55"] Oct 30 19:02:17.933525 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:17.933495 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2mvg\" (UniqueName: \"kubernetes.io/projected/69f6f804-4aba-432e-84d8-b969dc6a91d6-kube-api-access-g2mvg\") pod \"test-curl-7414-f00c55\" (UID: \"69f6f804-4aba-432e-84d8-b969dc6a91d6\") " pod="opendatahub/test-curl-7414-f00c55" Oct 30 19:02:18.034730 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:18.034648 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2mvg\" (UniqueName: \"kubernetes.io/projected/69f6f804-4aba-432e-84d8-b969dc6a91d6-kube-api-access-g2mvg\") pod \"test-curl-7414-f00c55\" (UID: \"69f6f804-4aba-432e-84d8-b969dc6a91d6\") " pod="opendatahub/test-curl-7414-f00c55" Oct 30 19:02:18.049366 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:18.049341 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2mvg\" (UniqueName: \"kubernetes.io/projected/69f6f804-4aba-432e-84d8-b969dc6a91d6-kube-api-access-g2mvg\") pod \"test-curl-7414-f00c55\" (UID: \"69f6f804-4aba-432e-84d8-b969dc6a91d6\") " pod="opendatahub/test-curl-7414-f00c55" Oct 30 19:02:18.082550 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:18.082521 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7414-f00c55" Oct 30 19:02:18.224282 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:18.224235 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:18.366002 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:18.365577 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7414-f00c55"] Oct 30 19:02:18.440310 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:18.440272 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7414-f00c55" event={"ID":"69f6f804-4aba-432e-84d8-b969dc6a91d6","Type":"ContainerStarted","Data":"2cd742b183be75fc033d1bcd7eeb9a59812040009e9bb55956ff9947a53d7402"} Oct 30 19:02:19.023420 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:19.023389 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:19.793931 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:19.793889 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:20.629827 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:20.629783 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:21.462667 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:21.462578 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7414-f00c55" event={"ID":"69f6f804-4aba-432e-84d8-b969dc6a91d6","Type":"ContainerStarted","Data":"6499e10b5268d23c7be42398915d075775df87b9125b2595b221706bd71690d3"} Oct 30 19:02:21.504497 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:21.504432 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7414-f00c55" podStartSLOduration=1.7068007 podStartE2EDuration="4.504414296s" totalImagesPullingTime="2.797613596s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:02:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:02:21.503403935 +0000 UTC m=+1709.032435073" watchObservedRunningTime="2026-10-30 19:02:21.504414296 +0000 UTC m=+1709.033445434" Oct 30 19:02:21.595295 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:21.595259 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:22.281993 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:22.281935 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7414-f00c55"] Oct 30 19:02:22.286175 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:22.286125 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7414-f00c55"] Oct 30 19:02:22.448710 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:22.448674 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7414-79ea93"] Oct 30 19:02:22.449547 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:22.449519 6960 cpu_manager.go:414] "containerMap: removing container" podUID="69f6f804-4aba-432e-84d8-b969dc6a91d6" containerName="test-curl-7414-f00c55" Oct 30 19:02:22.449547 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:22.449540 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69f6f804-4aba-432e-84d8-b969dc6a91d6" containerName="test-curl-7414-f00c55" Oct 30 19:02:22.449779 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:22.449765 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="69f6f804-4aba-432e-84d8-b969dc6a91d6" containerName="test-curl-7414-f00c55" Oct 30 19:02:22.453606 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:22.453583 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7414-79ea93" Oct 30 19:02:22.464100 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:22.464070 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7414-79ea93"] Oct 30 19:02:22.498238 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:22.498194 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5l86\" (UniqueName: \"kubernetes.io/projected/5dc2ff2c-3070-4256-b1c4-121699fefe9d-kube-api-access-x5l86\") pod \"test-curl-7414-79ea93\" (UID: \"5dc2ff2c-3070-4256-b1c4-121699fefe9d\") " pod="opendatahub/test-curl-7414-79ea93" Oct 30 19:02:22.557092 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:22.557061 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:22.600840 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:22.600802 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5l86\" (UniqueName: \"kubernetes.io/projected/5dc2ff2c-3070-4256-b1c4-121699fefe9d-kube-api-access-x5l86\") pod \"test-curl-7414-79ea93\" (UID: \"5dc2ff2c-3070-4256-b1c4-121699fefe9d\") " pod="opendatahub/test-curl-7414-79ea93" Oct 30 19:02:22.612754 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:22.612723 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5l86\" (UniqueName: \"kubernetes.io/projected/5dc2ff2c-3070-4256-b1c4-121699fefe9d-kube-api-access-x5l86\") pod \"test-curl-7414-79ea93\" (UID: \"5dc2ff2c-3070-4256-b1c4-121699fefe9d\") " pod="opendatahub/test-curl-7414-79ea93" Oct 30 19:02:22.768104 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:22.768068 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7414-79ea93" Oct 30 19:02:23.045891 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:23.045866 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7414-79ea93"] Oct 30 19:02:23.047310 ip-10-0-140-163 kubenswrapper[6960]: W1030 19:02:23.047287 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5dc2ff2c_3070_4256_b1c4_121699fefe9d.slice/crio-acc382a83a791c5085c8e50f5cb0fb60a1c773a2b4c8013a48662ce13e248b3a WatchSource:0}: Error finding container acc382a83a791c5085c8e50f5cb0fb60a1c773a2b4c8013a48662ce13e248b3a: Status 404 returned error can't find the container with id acc382a83a791c5085c8e50f5cb0fb60a1c773a2b4c8013a48662ce13e248b3a Oct 30 19:02:23.401525 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:23.401430 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:23.481222 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:23.481118 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7414-f00c55" podUID="69f6f804-4aba-432e-84d8-b969dc6a91d6" containerName="test-curl-7414-f00c55" containerID="cri-o://6499e10b5268d23c7be42398915d075775df87b9125b2595b221706bd71690d3" gracePeriod=2 Oct 30 19:02:23.482120 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:23.482098 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7414-79ea93" event={"ID":"5dc2ff2c-3070-4256-b1c4-121699fefe9d","Type":"ContainerStarted","Data":"313de54e63d49a47641dd7aa31f28c939f6468c72e4b2bced5ead706fad02b96"} Oct 30 19:02:23.482202 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:23.482135 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7414-79ea93" event={"ID":"5dc2ff2c-3070-4256-b1c4-121699fefe9d","Type":"ContainerStarted","Data":"acc382a83a791c5085c8e50f5cb0fb60a1c773a2b4c8013a48662ce13e248b3a"} Oct 30 19:02:23.498704 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:23.498648 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7414-79ea93" podStartSLOduration=1.4986342910000001 podStartE2EDuration="1.498634291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:02:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:02:23.496858336 +0000 UTC m=+1711.025889474" watchObservedRunningTime="2026-10-30 19:02:23.498634291 +0000 UTC m=+1711.027665482" Oct 30 19:02:23.935922 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:02:23.935880 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 19:02:24.217186 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:24.217089 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:24.252979 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:24.252941 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7414-79ea93"] Oct 30 19:02:24.259424 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:24.259392 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7414-79ea93"] Oct 30 19:02:24.941840 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:24.941805 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:25.051375 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:25.051341 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:25.428891 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:25.428842 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:25.562221 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:25.562185 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7414-79ea93" podUID="5dc2ff2c-3070-4256-b1c4-121699fefe9d" containerName="test-curl-7414-79ea93" containerID="cri-o://313de54e63d49a47641dd7aa31f28c939f6468c72e4b2bced5ead706fad02b96" gracePeriod=2 Oct 30 19:02:25.818042 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:25.818012 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:25.900708 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:25.900678 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7414-f00c55" Oct 30 19:02:26.069680 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:26.069651 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69f6f804-4aba-432e-84d8-b969dc6a91d6-kube-api-access-g2mvg\" (UniqueName: \"kubernetes.io/projected/69f6f804-4aba-432e-84d8-b969dc6a91d6-kube-api-access-g2mvg\") pod \"69f6f804-4aba-432e-84d8-b969dc6a91d6\" (UID: \"69f6f804-4aba-432e-84d8-b969dc6a91d6\") " Oct 30 19:02:26.072067 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:26.072040 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69f6f804-4aba-432e-84d8-b969dc6a91d6-kube-api-access-g2mvg" pod "69f6f804-4aba-432e-84d8-b969dc6a91d6" (UID: "69f6f804-4aba-432e-84d8-b969dc6a91d6"). InnerVolumeSpecName "kube-api-access-g2mvg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:02:26.172553 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:26.172430 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2mvg\" (UniqueName: \"kubernetes.io/projected/69f6f804-4aba-432e-84d8-b969dc6a91d6-kube-api-access-g2mvg\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:02:26.233540 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:26.233502 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:26.457974 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:26.457875 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:26.572130 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:26.572099 6960 generic.go:356] "Generic (PLEG): container finished" podID="69f6f804-4aba-432e-84d8-b969dc6a91d6" containerID="6499e10b5268d23c7be42398915d075775df87b9125b2595b221706bd71690d3" exitCode=137 Oct 30 19:02:26.572345 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:26.572192 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7414-f00c55" Oct 30 19:02:26.572345 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:26.572193 6960 scope.go:122] "RemoveContainer" containerID="6499e10b5268d23c7be42398915d075775df87b9125b2595b221706bd71690d3" Oct 30 19:02:26.583181 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:26.583118 6960 scope.go:122] "RemoveContainer" containerID="6499e10b5268d23c7be42398915d075775df87b9125b2595b221706bd71690d3" Oct 30 19:02:26.583512 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:02:26.583495 6960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6499e10b5268d23c7be42398915d075775df87b9125b2595b221706bd71690d3\": container with ID starting with 6499e10b5268d23c7be42398915d075775df87b9125b2595b221706bd71690d3 not found: ID does not exist" containerID="6499e10b5268d23c7be42398915d075775df87b9125b2595b221706bd71690d3" Oct 30 19:02:26.583561 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:26.583521 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6499e10b5268d23c7be42398915d075775df87b9125b2595b221706bd71690d3"} err="failed to get container status \"6499e10b5268d23c7be42398915d075775df87b9125b2595b221706bd71690d3\": rpc error: code = NotFound desc = could not find container \"6499e10b5268d23c7be42398915d075775df87b9125b2595b221706bd71690d3\": container with ID starting with 6499e10b5268d23c7be42398915d075775df87b9125b2595b221706bd71690d3 not found: ID does not exist" Oct 30 19:02:26.584841 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:02:26.584816 6960 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7414-f00c55\" is forbidden: User \"system:node:ip-10-0-140-163.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-163.us-east-2.compute.internal' and this object" podUID="69f6f804-4aba-432e-84d8-b969dc6a91d6" pod="opendatahub/test-curl-7414-f00c55" Oct 30 19:02:26.592735 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:26.592709 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69f6f804-4aba-432e-84d8-b969dc6a91d6" path="/var/lib/kubelet/pods/69f6f804-4aba-432e-84d8-b969dc6a91d6/volumes" Oct 30 19:02:27.008136 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:27.008095 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:27.788051 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:27.788025 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7414-79ea93" Oct 30 19:02:27.796524 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:27.796501 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dc2ff2c-3070-4256-b1c4-121699fefe9d-kube-api-access-x5l86\" (UniqueName: \"kubernetes.io/projected/5dc2ff2c-3070-4256-b1c4-121699fefe9d-kube-api-access-x5l86\") pod \"5dc2ff2c-3070-4256-b1c4-121699fefe9d\" (UID: \"5dc2ff2c-3070-4256-b1c4-121699fefe9d\") " Oct 30 19:02:27.798594 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:27.798571 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dc2ff2c-3070-4256-b1c4-121699fefe9d-kube-api-access-x5l86" pod "5dc2ff2c-3070-4256-b1c4-121699fefe9d" (UID: "5dc2ff2c-3070-4256-b1c4-121699fefe9d"). InnerVolumeSpecName "kube-api-access-x5l86". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:02:27.817106 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:27.817081 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:27.899238 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:27.899116 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5l86\" (UniqueName: \"kubernetes.io/projected/5dc2ff2c-3070-4256-b1c4-121699fefe9d-kube-api-access-x5l86\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:02:28.591333 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:28.591300 6960 generic.go:356] "Generic (PLEG): container finished" podID="5dc2ff2c-3070-4256-b1c4-121699fefe9d" containerID="313de54e63d49a47641dd7aa31f28c939f6468c72e4b2bced5ead706fad02b96" exitCode=137 Oct 30 19:02:28.591536 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:28.591393 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7414-79ea93" Oct 30 19:02:28.593867 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:28.593840 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5dc2ff2c-3070-4256-b1c4-121699fefe9d" path="/var/lib/kubelet/pods/5dc2ff2c-3070-4256-b1c4-121699fefe9d/volumes" Oct 30 19:02:28.594320 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:28.594304 6960 scope.go:122] "RemoveContainer" containerID="313de54e63d49a47641dd7aa31f28c939f6468c72e4b2bced5ead706fad02b96" Oct 30 19:02:28.605217 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:28.605194 6960 scope.go:122] "RemoveContainer" containerID="313de54e63d49a47641dd7aa31f28c939f6468c72e4b2bced5ead706fad02b96" Oct 30 19:02:28.605493 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:02:28.605476 6960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"313de54e63d49a47641dd7aa31f28c939f6468c72e4b2bced5ead706fad02b96\": container with ID starting with 313de54e63d49a47641dd7aa31f28c939f6468c72e4b2bced5ead706fad02b96 not found: ID does not exist" containerID="313de54e63d49a47641dd7aa31f28c939f6468c72e4b2bced5ead706fad02b96" Oct 30 19:02:28.605585 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:28.605501 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"313de54e63d49a47641dd7aa31f28c939f6468c72e4b2bced5ead706fad02b96"} err="failed to get container status \"313de54e63d49a47641dd7aa31f28c939f6468c72e4b2bced5ead706fad02b96\": rpc error: code = NotFound desc = could not find container \"313de54e63d49a47641dd7aa31f28c939f6468c72e4b2bced5ead706fad02b96\": container with ID starting with 313de54e63d49a47641dd7aa31f28c939f6468c72e4b2bced5ead706fad02b96 not found: ID does not exist" Oct 30 19:02:28.613160 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:28.613126 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:29.523856 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:29.523816 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:29.802390 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:29.802311 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:30.030473 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:30.030445 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:30.722789 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:30.722735 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:31.052348 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:31.052266 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:31.079711 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:31.079678 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e1e1ff-5f47bcf47f-2wj2p"] Oct 30 19:02:31.080303 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:31.080288 6960 cpu_manager.go:414] "containerMap: removing container" podUID="5dc2ff2c-3070-4256-b1c4-121699fefe9d" containerName="test-curl-7414-79ea93" Oct 30 19:02:31.080303 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:31.080303 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5dc2ff2c-3070-4256-b1c4-121699fefe9d" containerName="test-curl-7414-79ea93" Oct 30 19:02:31.080452 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:31.080443 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="5dc2ff2c-3070-4256-b1c4-121699fefe9d" containerName="test-curl-7414-79ea93" Oct 30 19:02:31.085236 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:31.085217 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e1e1ff-5f47bcf47f-2wj2p" Oct 30 19:02:31.093820 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:31.093646 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-e1e1ff\"" Oct 30 19:02:31.094446 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:31.094425 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e1e1ff-5f47bcf47f-2wj2p"] Oct 30 19:02:31.143761 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:31.143714 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjb99\" (UniqueName: \"kubernetes.io/projected/ccb27153-3ebf-4eaa-80a1-e460de5d78d0-kube-api-access-tjb99\") pod \"payload-pre-processing-e2e-shared-a-w2-e1e1ff-5f47bcf47f-2wj2p\" (UID: \"ccb27153-3ebf-4eaa-80a1-e460de5d78d0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e1e1ff-5f47bcf47f-2wj2p" Oct 30 19:02:31.143951 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:31.143869 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccb27153-3ebf-4eaa-80a1-e460de5d78d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-e1e1ff-5f47bcf47f-2wj2p\" (UID: \"ccb27153-3ebf-4eaa-80a1-e460de5d78d0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e1e1ff-5f47bcf47f-2wj2p" Oct 30 19:02:31.153651 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:31.153627 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-e1e1ff-848c46dccb-rbs5r"] Oct 30 19:02:31.157958 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:31.157940 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e1e1ff-848c46dccb-rbs5r" Oct 30 19:02:31.160800 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:31.160775 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:31.167182 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:31.167123 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-e1e1ff-dockercfg-lgnw6\"" Oct 30 19:02:31.170639 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:31.170621 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-e1e1ff-848c46dccb-rbs5r"] Oct 30 19:02:31.246210 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:31.246122 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjb99\" (UniqueName: \"kubernetes.io/projected/ccb27153-3ebf-4eaa-80a1-e460de5d78d0-kube-api-access-tjb99\") pod \"payload-pre-processing-e2e-shared-a-w2-e1e1ff-5f47bcf47f-2wj2p\" (UID: \"ccb27153-3ebf-4eaa-80a1-e460de5d78d0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e1e1ff-5f47bcf47f-2wj2p" Oct 30 19:02:31.285168 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:31.278525 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccb27153-3ebf-4eaa-80a1-e460de5d78d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-e1e1ff-5f47bcf47f-2wj2p\" (UID: \"ccb27153-3ebf-4eaa-80a1-e460de5d78d0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e1e1ff-5f47bcf47f-2wj2p" Oct 30 19:02:31.285168 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:31.279789 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccb27153-3ebf-4eaa-80a1-e460de5d78d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-e1e1ff-5f47bcf47f-2wj2p\" (UID: \"ccb27153-3ebf-4eaa-80a1-e460de5d78d0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e1e1ff-5f47bcf47f-2wj2p" Oct 30 19:02:31.320483 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:31.304383 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjb99\" (UniqueName: \"kubernetes.io/projected/ccb27153-3ebf-4eaa-80a1-e460de5d78d0-kube-api-access-tjb99\") pod \"payload-pre-processing-e2e-shared-a-w2-e1e1ff-5f47bcf47f-2wj2p\" (UID: \"ccb27153-3ebf-4eaa-80a1-e460de5d78d0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e1e1ff-5f47bcf47f-2wj2p" Oct 30 19:02:31.380999 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:31.380961 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbc9b\" (UniqueName: \"kubernetes.io/projected/6804806b-3a5b-425d-9ae7-55bda5f68a4e-kube-api-access-nbc9b\") pod \"payload-processing-e2e-shared-a-w2-e1e1ff-848c46dccb-rbs5r\" (UID: \"6804806b-3a5b-425d-9ae7-55bda5f68a4e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e1e1ff-848c46dccb-rbs5r" Oct 30 19:02:31.381360 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:31.381338 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6804806b-3a5b-425d-9ae7-55bda5f68a4e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-e1e1ff-848c46dccb-rbs5r\" (UID: \"6804806b-3a5b-425d-9ae7-55bda5f68a4e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e1e1ff-848c46dccb-rbs5r" Oct 30 19:02:31.401630 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:31.401594 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e1e1ff-5f47bcf47f-2wj2p" Oct 30 19:02:31.483818 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:31.483476 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbc9b\" (UniqueName: \"kubernetes.io/projected/6804806b-3a5b-425d-9ae7-55bda5f68a4e-kube-api-access-nbc9b\") pod \"payload-processing-e2e-shared-a-w2-e1e1ff-848c46dccb-rbs5r\" (UID: \"6804806b-3a5b-425d-9ae7-55bda5f68a4e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e1e1ff-848c46dccb-rbs5r" Oct 30 19:02:31.483818 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:31.483532 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6804806b-3a5b-425d-9ae7-55bda5f68a4e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-e1e1ff-848c46dccb-rbs5r\" (UID: \"6804806b-3a5b-425d-9ae7-55bda5f68a4e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e1e1ff-848c46dccb-rbs5r" Oct 30 19:02:31.485347 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:31.484540 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6804806b-3a5b-425d-9ae7-55bda5f68a4e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-e1e1ff-848c46dccb-rbs5r\" (UID: \"6804806b-3a5b-425d-9ae7-55bda5f68a4e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e1e1ff-848c46dccb-rbs5r" Oct 30 19:02:31.498244 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:31.497702 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbc9b\" (UniqueName: \"kubernetes.io/projected/6804806b-3a5b-425d-9ae7-55bda5f68a4e-kube-api-access-nbc9b\") pod \"payload-processing-e2e-shared-a-w2-e1e1ff-848c46dccb-rbs5r\" (UID: \"6804806b-3a5b-425d-9ae7-55bda5f68a4e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e1e1ff-848c46dccb-rbs5r" Oct 30 19:02:31.696681 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:31.696652 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e1e1ff-5f47bcf47f-2wj2p"] Oct 30 19:02:31.697834 ip-10-0-140-163 kubenswrapper[6960]: W1030 19:02:31.697811 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccb27153_3ebf_4eaa_80a1_e460de5d78d0.slice/crio-ba951fb17792ed3ce4349d99d77bb240fb290522c42911bd9256c15a96166e1e WatchSource:0}: Error finding container ba951fb17792ed3ce4349d99d77bb240fb290522c42911bd9256c15a96166e1e: Status 404 returned error can't find the container with id ba951fb17792ed3ce4349d99d77bb240fb290522c42911bd9256c15a96166e1e Oct 30 19:02:31.731515 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:31.731486 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:31.775453 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:31.775427 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e1e1ff-848c46dccb-rbs5r" Oct 30 19:02:32.072412 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:32.072388 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-e1e1ff-848c46dccb-rbs5r"] Oct 30 19:02:32.072966 ip-10-0-140-163 kubenswrapper[6960]: W1030 19:02:32.072944 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6804806b_3a5b_425d_9ae7_55bda5f68a4e.slice/crio-145754ca7a44a4a136c04407d8b850a2d2c78d1dcdea8432902d9f081050a75d WatchSource:0}: Error finding container 145754ca7a44a4a136c04407d8b850a2d2c78d1dcdea8432902d9f081050a75d: Status 404 returned error can't find the container with id 145754ca7a44a4a136c04407d8b850a2d2c78d1dcdea8432902d9f081050a75d Oct 30 19:02:32.621367 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:32.621332 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:32.622402 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:32.622369 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e1e1ff-848c46dccb-rbs5r" event={"ID":"6804806b-3a5b-425d-9ae7-55bda5f68a4e","Type":"ContainerStarted","Data":"4b8015039308c712a31052f4da61fc06559002366e60f307cde54bf586651785"} Oct 30 19:02:32.622607 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:32.622426 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e1e1ff-848c46dccb-rbs5r" event={"ID":"6804806b-3a5b-425d-9ae7-55bda5f68a4e","Type":"ContainerStarted","Data":"145754ca7a44a4a136c04407d8b850a2d2c78d1dcdea8432902d9f081050a75d"} Oct 30 19:02:32.622607 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:32.622602 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e1e1ff-848c46dccb-rbs5r" Oct 30 19:02:32.624332 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:32.624311 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e1e1ff-5f47bcf47f-2wj2p" event={"ID":"ccb27153-3ebf-4eaa-80a1-e460de5d78d0","Type":"ContainerStarted","Data":"0c6d78f830007a990d4be79959873f72c8a088d132f06576bc19821e8c5bf572"} Oct 30 19:02:32.624410 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:32.624349 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e1e1ff-5f47bcf47f-2wj2p" Oct 30 19:02:32.624410 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:32.624365 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e1e1ff-5f47bcf47f-2wj2p" event={"ID":"ccb27153-3ebf-4eaa-80a1-e460de5d78d0","Type":"ContainerStarted","Data":"ba951fb17792ed3ce4349d99d77bb240fb290522c42911bd9256c15a96166e1e"} Oct 30 19:02:32.645780 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:32.645741 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e1e1ff-848c46dccb-rbs5r" podStartSLOduration=1.64572356 podStartE2EDuration="1.64572356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:02:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:02:32.642336872 +0000 UTC m=+1720.171368010" watchObservedRunningTime="2026-10-30 19:02:32.64572356 +0000 UTC m=+1720.174754697" Oct 30 19:02:32.668174 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:32.667722 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e1e1ff-5f47bcf47f-2wj2p" podStartSLOduration=1.667699671 podStartE2EDuration="1.667699671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:02:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:02:32.665892741 +0000 UTC m=+1720.194923918" watchObservedRunningTime="2026-10-30 19:02:32.667699671 +0000 UTC m=+1720.196730824" Oct 30 19:02:33.405125 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:33.405048 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:34.235212 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:34.235177 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:35.021884 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:35.021849 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:35.923277 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:35.923231 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:36.731296 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:36.731258 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:37.713471 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:37.713438 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:38.578462 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:38.578431 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:39.617131 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:39.617082 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:40.448161 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:40.448110 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:41.245879 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:41.245828 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e1e1ff-5d6f9c8b5-xsp5m"] Oct 30 19:02:41.251835 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:41.251813 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e1e1ff-5d6f9c8b5-xsp5m" Oct 30 19:02:41.263815 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:41.263777 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-e1e1ff-serving-cert\"" Oct 30 19:02:41.269301 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:41.269271 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e1e1ff-5d6f9c8b5-xsp5m"] Oct 30 19:02:41.314060 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:41.314026 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48c4bb82-3969-4e2d-a303-ef614827b953-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-e1e1ff-5d6f9c8b5-xsp5m\" (UID: \"48c4bb82-3969-4e2d-a303-ef614827b953\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e1e1ff-5d6f9c8b5-xsp5m" Oct 30 19:02:41.314401 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:41.314117 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7wrw\" (UniqueName: \"kubernetes.io/projected/48c4bb82-3969-4e2d-a303-ef614827b953-kube-api-access-r7wrw\") pod \"maas-api-e2e-shared-a-w2-e1e1ff-5d6f9c8b5-xsp5m\" (UID: \"48c4bb82-3969-4e2d-a303-ef614827b953\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e1e1ff-5d6f9c8b5-xsp5m" Oct 30 19:02:41.314401 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:41.314184 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48c4bb82-3969-4e2d-a303-ef614827b953-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-e1e1ff-5d6f9c8b5-xsp5m\" (UID: \"48c4bb82-3969-4e2d-a303-ef614827b953\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e1e1ff-5d6f9c8b5-xsp5m" Oct 30 19:02:41.314401 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:41.314218 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48c4bb82-3969-4e2d-a303-ef614827b953-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-e1e1ff-5d6f9c8b5-xsp5m\" (UID: \"48c4bb82-3969-4e2d-a303-ef614827b953\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e1e1ff-5d6f9c8b5-xsp5m" Oct 30 19:02:41.314678 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:41.314527 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:41.419196 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:41.417893 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48c4bb82-3969-4e2d-a303-ef614827b953-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-e1e1ff-5d6f9c8b5-xsp5m\" (UID: \"48c4bb82-3969-4e2d-a303-ef614827b953\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e1e1ff-5d6f9c8b5-xsp5m" Oct 30 19:02:41.419196 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:41.418030 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7wrw\" (UniqueName: \"kubernetes.io/projected/48c4bb82-3969-4e2d-a303-ef614827b953-kube-api-access-r7wrw\") pod \"maas-api-e2e-shared-a-w2-e1e1ff-5d6f9c8b5-xsp5m\" (UID: \"48c4bb82-3969-4e2d-a303-ef614827b953\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e1e1ff-5d6f9c8b5-xsp5m" Oct 30 19:02:41.419196 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:41.418097 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48c4bb82-3969-4e2d-a303-ef614827b953-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-e1e1ff-5d6f9c8b5-xsp5m\" (UID: \"48c4bb82-3969-4e2d-a303-ef614827b953\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e1e1ff-5d6f9c8b5-xsp5m" Oct 30 19:02:41.419196 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:41.418133 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48c4bb82-3969-4e2d-a303-ef614827b953-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-e1e1ff-5d6f9c8b5-xsp5m\" (UID: \"48c4bb82-3969-4e2d-a303-ef614827b953\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e1e1ff-5d6f9c8b5-xsp5m" Oct 30 19:02:41.419196 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:02:41.418332 6960 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e1e1ff-serving-cert: secret "maas-api-e2e-shared-a-w2-e1e1ff-serving-cert" not found Oct 30 19:02:41.419196 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:02:41.418391 6960 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48c4bb82-3969-4e2d-a303-ef614827b953-maas-api-tls podName:48c4bb82-3969-4e2d-a303-ef614827b953 nodeName:}" failed. No retries permitted until 2026-10-30 19:02:41.918370998 +0000 UTC m=+1729.447402128 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/48c4bb82-3969-4e2d-a303-ef614827b953-maas-api-tls") pod "maas-api-e2e-shared-a-w2-e1e1ff-5d6f9c8b5-xsp5m" (UID: "48c4bb82-3969-4e2d-a303-ef614827b953") : secret "maas-api-e2e-shared-a-w2-e1e1ff-serving-cert" not found Oct 30 19:02:41.419196 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:41.418817 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48c4bb82-3969-4e2d-a303-ef614827b953-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-e1e1ff-5d6f9c8b5-xsp5m\" (UID: \"48c4bb82-3969-4e2d-a303-ef614827b953\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e1e1ff-5d6f9c8b5-xsp5m" Oct 30 19:02:41.421964 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:41.421936 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48c4bb82-3969-4e2d-a303-ef614827b953-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-e1e1ff-5d6f9c8b5-xsp5m\" (UID: \"48c4bb82-3969-4e2d-a303-ef614827b953\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e1e1ff-5d6f9c8b5-xsp5m" Oct 30 19:02:41.439698 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:41.439666 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7wrw\" (UniqueName: \"kubernetes.io/projected/48c4bb82-3969-4e2d-a303-ef614827b953-kube-api-access-r7wrw\") pod \"maas-api-e2e-shared-a-w2-e1e1ff-5d6f9c8b5-xsp5m\" (UID: \"48c4bb82-3969-4e2d-a303-ef614827b953\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e1e1ff-5d6f9c8b5-xsp5m" Oct 30 19:02:41.443719 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:41.443690 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:41.599408 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:41.599376 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:41.926991 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:41.926188 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48c4bb82-3969-4e2d-a303-ef614827b953-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-e1e1ff-5d6f9c8b5-xsp5m\" (UID: \"48c4bb82-3969-4e2d-a303-ef614827b953\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e1e1ff-5d6f9c8b5-xsp5m" Oct 30 19:02:41.929850 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:41.929819 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48c4bb82-3969-4e2d-a303-ef614827b953-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-e1e1ff-5d6f9c8b5-xsp5m\" (UID: \"48c4bb82-3969-4e2d-a303-ef614827b953\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e1e1ff-5d6f9c8b5-xsp5m" Oct 30 19:02:42.177592 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:42.177507 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e1e1ff-5d6f9c8b5-xsp5m" Oct 30 19:02:42.489539 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:42.489514 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e1e1ff-5d6f9c8b5-xsp5m"] Oct 30 19:02:42.490479 ip-10-0-140-163 kubenswrapper[6960]: W1030 19:02:42.490454 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48c4bb82_3969_4e2d_a303_ef614827b953.slice/crio-8ae0e2b7ed57d6e0c706e9a14e06c386ee51651e9c227b3595c4709fbc1a45af WatchSource:0}: Error finding container 8ae0e2b7ed57d6e0c706e9a14e06c386ee51651e9c227b3595c4709fbc1a45af: Status 404 returned error can't find the container with id 8ae0e2b7ed57d6e0c706e9a14e06c386ee51651e9c227b3595c4709fbc1a45af Oct 30 19:02:42.496341 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:42.496315 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:42.691388 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:42.691270 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e1e1ff-5d6f9c8b5-xsp5m" event={"ID":"48c4bb82-3969-4e2d-a303-ef614827b953","Type":"ContainerStarted","Data":"8ae0e2b7ed57d6e0c706e9a14e06c386ee51651e9c227b3595c4709fbc1a45af"} Oct 30 19:02:43.311870 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:43.311838 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:43.633016 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:43.632948 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e1e1ff-848c46dccb-rbs5r" Oct 30 19:02:43.633016 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:43.633009 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e1e1ff-5f47bcf47f-2wj2p" Oct 30 19:02:43.700959 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:43.700920 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e1e1ff-5d6f9c8b5-xsp5m" event={"ID":"48c4bb82-3969-4e2d-a303-ef614827b953","Type":"ContainerStarted","Data":"fd345babda3c5c2a832a518ff73349085977ec2afb294ad7baa56754aabab6eb"} Oct 30 19:02:43.701183 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:43.701068 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e1e1ff-5d6f9c8b5-xsp5m" Oct 30 19:02:43.724030 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:43.723978 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e1e1ff-5d6f9c8b5-xsp5m" podStartSLOduration=2.250121667 podStartE2EDuration="2.723963946s" totalImagesPullingTime="473.842279ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:02:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:02:43.720906249 +0000 UTC m=+1731.249937386" watchObservedRunningTime="2026-10-30 19:02:43.723963946 +0000 UTC m=+1731.252995083" Oct 30 19:02:44.172286 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:44.172238 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:45.019661 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:45.019619 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:45.832746 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:45.832704 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:46.920897 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:46.920859 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:47.960039 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:47.959907 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:48.847054 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:48.847014 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:49.748170 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:49.741319 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e1e1ff-5d6f9c8b5-xsp5m" Oct 30 19:02:49.845112 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:49.844801 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:50.899651 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:50.899606 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:51.955408 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:51.955373 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:52.502848 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:52.502792 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:53.060063 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:53.059990 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:53.920439 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:53.920400 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:53.951924 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:02:53.951888 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 19:02:54.971617 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:54.971580 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:55.770057 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:55.770021 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:56.750359 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:56.750326 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:57.619988 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:57.619955 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:58.512421 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:58.512386 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:02:59.402057 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:02:59.402026 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:00.158749 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:00.158712 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:00.263492 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:00.263453 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:01.044325 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:01.044294 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:01.696464 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:01.696427 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:01.817114 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:01.817086 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:01.940529 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:01.940491 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:02.290983 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:02.290948 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:02.546729 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:02.546647 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:03.200874 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:03.200842 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:04.394715 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:04.394677 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:05.847488 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:05.847455 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:06.521011 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:06.520976 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:06.686523 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:06.686474 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:07.180441 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:07.180389 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:07.794138 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:07.794097 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:07.832937 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:07.832913 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-b0e009-bfd774574-dl2n4"] Oct 30 19:03:07.841005 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:07.840986 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b0e009-bfd774574-dl2n4" Oct 30 19:03:07.861685 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:07.861667 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-b0e009-dockercfg-hghcj\"" Oct 30 19:03:07.863024 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:07.863004 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-b0e009\"" Oct 30 19:03:07.866513 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:07.866490 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-b0e009-bfd774574-dl2n4"] Oct 30 19:03:07.887759 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:07.887727 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b0e009-bb65989f9-m7q2f"] Oct 30 19:03:07.893176 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:07.893129 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b0e009-bb65989f9-m7q2f" Oct 30 19:03:07.923200 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:07.923131 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b0e009-bb65989f9-m7q2f"] Oct 30 19:03:07.964791 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:07.964758 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3541c21-2190-4ba9-a2aa-286582db6f34-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-b0e009-bfd774574-dl2n4\" (UID: \"f3541c21-2190-4ba9-a2aa-286582db6f34\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b0e009-bfd774574-dl2n4" Oct 30 19:03:07.972987 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:07.972934 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fgps\" (UniqueName: \"kubernetes.io/projected/f3541c21-2190-4ba9-a2aa-286582db6f34-kube-api-access-8fgps\") pod \"payload-processing-e2e-shared-b-w2-b0e009-bfd774574-dl2n4\" (UID: \"f3541c21-2190-4ba9-a2aa-286582db6f34\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b0e009-bfd774574-dl2n4" Oct 30 19:03:08.011356 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:08.011031 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:08.077171 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:08.076132 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3541c21-2190-4ba9-a2aa-286582db6f34-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-b0e009-bfd774574-dl2n4\" (UID: \"f3541c21-2190-4ba9-a2aa-286582db6f34\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b0e009-bfd774574-dl2n4" Oct 30 19:03:08.077171 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:08.076244 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ebea224-1a00-496f-a41e-1d7f2a294a5b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-b0e009-bb65989f9-m7q2f\" (UID: \"6ebea224-1a00-496f-a41e-1d7f2a294a5b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b0e009-bb65989f9-m7q2f" Oct 30 19:03:08.077171 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:08.076371 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fgps\" (UniqueName: \"kubernetes.io/projected/f3541c21-2190-4ba9-a2aa-286582db6f34-kube-api-access-8fgps\") pod \"payload-processing-e2e-shared-b-w2-b0e009-bfd774574-dl2n4\" (UID: \"f3541c21-2190-4ba9-a2aa-286582db6f34\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b0e009-bfd774574-dl2n4" Oct 30 19:03:08.077171 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:08.076437 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmmgh\" (UniqueName: \"kubernetes.io/projected/6ebea224-1a00-496f-a41e-1d7f2a294a5b-kube-api-access-nmmgh\") pod \"payload-pre-processing-e2e-shared-b-w2-b0e009-bb65989f9-m7q2f\" (UID: \"6ebea224-1a00-496f-a41e-1d7f2a294a5b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b0e009-bb65989f9-m7q2f" Oct 30 19:03:08.077754 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:08.077726 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3541c21-2190-4ba9-a2aa-286582db6f34-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-b0e009-bfd774574-dl2n4\" (UID: \"f3541c21-2190-4ba9-a2aa-286582db6f34\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b0e009-bfd774574-dl2n4" Oct 30 19:03:08.130826 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:08.127823 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fgps\" (UniqueName: \"kubernetes.io/projected/f3541c21-2190-4ba9-a2aa-286582db6f34-kube-api-access-8fgps\") pod \"payload-processing-e2e-shared-b-w2-b0e009-bfd774574-dl2n4\" (UID: \"f3541c21-2190-4ba9-a2aa-286582db6f34\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b0e009-bfd774574-dl2n4" Oct 30 19:03:08.155593 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:08.155565 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b0e009-bfd774574-dl2n4" Oct 30 19:03:08.180254 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:08.179864 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmmgh\" (UniqueName: \"kubernetes.io/projected/6ebea224-1a00-496f-a41e-1d7f2a294a5b-kube-api-access-nmmgh\") pod \"payload-pre-processing-e2e-shared-b-w2-b0e009-bb65989f9-m7q2f\" (UID: \"6ebea224-1a00-496f-a41e-1d7f2a294a5b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b0e009-bb65989f9-m7q2f" Oct 30 19:03:08.180464 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:08.180264 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ebea224-1a00-496f-a41e-1d7f2a294a5b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-b0e009-bb65989f9-m7q2f\" (UID: \"6ebea224-1a00-496f-a41e-1d7f2a294a5b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b0e009-bb65989f9-m7q2f" Oct 30 19:03:08.181010 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:08.180992 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ebea224-1a00-496f-a41e-1d7f2a294a5b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-b0e009-bb65989f9-m7q2f\" (UID: \"6ebea224-1a00-496f-a41e-1d7f2a294a5b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b0e009-bb65989f9-m7q2f" Oct 30 19:03:08.200428 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:08.200376 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmmgh\" (UniqueName: \"kubernetes.io/projected/6ebea224-1a00-496f-a41e-1d7f2a294a5b-kube-api-access-nmmgh\") pod \"payload-pre-processing-e2e-shared-b-w2-b0e009-bb65989f9-m7q2f\" (UID: \"6ebea224-1a00-496f-a41e-1d7f2a294a5b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b0e009-bb65989f9-m7q2f" Oct 30 19:03:08.208847 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:08.208811 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b0e009-bb65989f9-m7q2f" Oct 30 19:03:08.483013 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:08.482987 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-b0e009-bfd774574-dl2n4"] Oct 30 19:03:08.484253 ip-10-0-140-163 kubenswrapper[6960]: W1030 19:03:08.484224 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3541c21_2190_4ba9_a2aa_286582db6f34.slice/crio-1215f5cbcf060ffb529686f3bab7c44d3579f5c34bd6c102e4c4cfc15016e1a4 WatchSource:0}: Error finding container 1215f5cbcf060ffb529686f3bab7c44d3579f5c34bd6c102e4c4cfc15016e1a4: Status 404 returned error can't find the container with id 1215f5cbcf060ffb529686f3bab7c44d3579f5c34bd6c102e4c4cfc15016e1a4 Oct 30 19:03:08.520439 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:08.520413 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:08.537121 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:08.537100 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b0e009-bb65989f9-m7q2f"] Oct 30 19:03:08.539986 ip-10-0-140-163 kubenswrapper[6960]: W1030 19:03:08.539958 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ebea224_1a00_496f_a41e_1d7f2a294a5b.slice/crio-019746db4dc6fb1ffb02be63955ffaafbe83fd18e29b1552911aedef6e15024c WatchSource:0}: Error finding container 019746db4dc6fb1ffb02be63955ffaafbe83fd18e29b1552911aedef6e15024c: Status 404 returned error can't find the container with id 019746db4dc6fb1ffb02be63955ffaafbe83fd18e29b1552911aedef6e15024c Oct 30 19:03:08.975230 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:08.975176 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b0e009-bb65989f9-m7q2f" event={"ID":"6ebea224-1a00-496f-a41e-1d7f2a294a5b","Type":"ContainerStarted","Data":"797f04cb0740af078454d1f763dbfcf7ebf5c88bd63029ccdd5bf0e4fbe1dba7"} Oct 30 19:03:08.975409 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:08.975285 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b0e009-bb65989f9-m7q2f" event={"ID":"6ebea224-1a00-496f-a41e-1d7f2a294a5b","Type":"ContainerStarted","Data":"019746db4dc6fb1ffb02be63955ffaafbe83fd18e29b1552911aedef6e15024c"} Oct 30 19:03:08.975511 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:08.975442 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b0e009-bb65989f9-m7q2f" Oct 30 19:03:08.976980 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:08.976953 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b0e009-bfd774574-dl2n4" event={"ID":"f3541c21-2190-4ba9-a2aa-286582db6f34","Type":"ContainerStarted","Data":"18b412ec06a44da429e002741338dc60551bb66300e93211144a06ae621d4680"} Oct 30 19:03:08.977095 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:08.977000 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b0e009-bfd774574-dl2n4" event={"ID":"f3541c21-2190-4ba9-a2aa-286582db6f34","Type":"ContainerStarted","Data":"1215f5cbcf060ffb529686f3bab7c44d3579f5c34bd6c102e4c4cfc15016e1a4"} Oct 30 19:03:08.977095 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:08.977037 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b0e009-bfd774574-dl2n4" Oct 30 19:03:09.037833 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:09.037764 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b0e009-bb65989f9-m7q2f" podStartSLOduration=2.03774593 podStartE2EDuration="2.03774593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:03:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:03:09.034650658 +0000 UTC m=+1756.563681797" watchObservedRunningTime="2026-10-30 19:03:09.03774593 +0000 UTC m=+1756.566777069" Oct 30 19:03:09.894733 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:09.894693 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:11.299544 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:11.299497 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:12.352461 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:12.352416 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:13.277512 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:13.277479 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:14.340114 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:14.340075 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:15.342377 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:15.342338 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:16.402034 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:16.401992 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:16.695857 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:16.695735 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b0e009-bfd774574-dl2n4" podStartSLOduration=9.695716159 podStartE2EDuration="9.695716159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:03:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:03:09.0687926 +0000 UTC m=+1756.597823739" watchObservedRunningTime="2026-10-30 19:03:16.695716159 +0000 UTC m=+1764.224747299" Oct 30 19:03:16.698125 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:16.698102 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-fln48"] Oct 30 19:03:16.702464 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:16.702446 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-fln48" Oct 30 19:03:16.707182 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:16.707162 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-c3a5b9-openshift-default-dockercfg-b8nm9\"" Oct 30 19:03:16.718917 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:16.718877 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-fln48"] Oct 30 19:03:16.823768 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:16.823733 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txjt7\" (UniqueName: \"kubernetes.io/projected/904f96ca-921a-4d2e-8245-46500d6b7d0a-kube-api-access-txjt7\") pod \"e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-fln48\" (UID: \"904f96ca-921a-4d2e-8245-46500d6b7d0a\") " pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-fln48" Oct 30 19:03:16.823964 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:16.823867 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/904f96ca-921a-4d2e-8245-46500d6b7d0a-workload-certs\") pod \"e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-fln48\" (UID: \"904f96ca-921a-4d2e-8245-46500d6b7d0a\") " pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-fln48" Oct 30 19:03:16.823964 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:16.823906 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/904f96ca-921a-4d2e-8245-46500d6b7d0a-istio-token\") pod \"e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-fln48\" (UID: \"904f96ca-921a-4d2e-8245-46500d6b7d0a\") " pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-fln48" Oct 30 19:03:16.823964 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:16.823929 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/904f96ca-921a-4d2e-8245-46500d6b7d0a-workload-socket\") pod \"e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-fln48\" (UID: \"904f96ca-921a-4d2e-8245-46500d6b7d0a\") " pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-fln48" Oct 30 19:03:16.823964 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:16.823945 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/904f96ca-921a-4d2e-8245-46500d6b7d0a-credential-socket\") pod \"e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-fln48\" (UID: \"904f96ca-921a-4d2e-8245-46500d6b7d0a\") " pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-fln48" Oct 30 19:03:16.824168 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:16.824008 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/904f96ca-921a-4d2e-8245-46500d6b7d0a-istio-data\") pod \"e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-fln48\" (UID: \"904f96ca-921a-4d2e-8245-46500d6b7d0a\") " pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-fln48" Oct 30 19:03:16.824210 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:16.824175 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/904f96ca-921a-4d2e-8245-46500d6b7d0a-istiod-ca-cert\") pod \"e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-fln48\" (UID: \"904f96ca-921a-4d2e-8245-46500d6b7d0a\") " pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-fln48" Oct 30 19:03:16.824325 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:16.824308 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/904f96ca-921a-4d2e-8245-46500d6b7d0a-istio-envoy\") pod \"e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-fln48\" (UID: \"904f96ca-921a-4d2e-8245-46500d6b7d0a\") " pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-fln48" Oct 30 19:03:16.824378 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:16.824366 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/904f96ca-921a-4d2e-8245-46500d6b7d0a-istio-podinfo\") pod \"e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-fln48\" (UID: \"904f96ca-921a-4d2e-8245-46500d6b7d0a\") " pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-fln48" Oct 30 19:03:16.925603 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:16.925570 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/904f96ca-921a-4d2e-8245-46500d6b7d0a-istiod-ca-cert\") pod \"e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-fln48\" (UID: \"904f96ca-921a-4d2e-8245-46500d6b7d0a\") " pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-fln48" Oct 30 19:03:16.925790 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:16.925684 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/904f96ca-921a-4d2e-8245-46500d6b7d0a-istio-envoy\") pod \"e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-fln48\" (UID: \"904f96ca-921a-4d2e-8245-46500d6b7d0a\") " pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-fln48" Oct 30 19:03:16.925790 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:16.925737 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/904f96ca-921a-4d2e-8245-46500d6b7d0a-istio-podinfo\") pod \"e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-fln48\" (UID: \"904f96ca-921a-4d2e-8245-46500d6b7d0a\") " pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-fln48" Oct 30 19:03:16.925908 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:16.925816 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txjt7\" (UniqueName: \"kubernetes.io/projected/904f96ca-921a-4d2e-8245-46500d6b7d0a-kube-api-access-txjt7\") pod \"e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-fln48\" (UID: \"904f96ca-921a-4d2e-8245-46500d6b7d0a\") " pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-fln48" Oct 30 19:03:16.925908 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:16.925879 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/904f96ca-921a-4d2e-8245-46500d6b7d0a-workload-certs\") pod \"e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-fln48\" (UID: \"904f96ca-921a-4d2e-8245-46500d6b7d0a\") " pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-fln48" Oct 30 19:03:16.926013 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:16.925910 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/904f96ca-921a-4d2e-8245-46500d6b7d0a-istio-token\") pod \"e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-fln48\" (UID: \"904f96ca-921a-4d2e-8245-46500d6b7d0a\") " pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-fln48" Oct 30 19:03:16.926013 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:16.925939 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/904f96ca-921a-4d2e-8245-46500d6b7d0a-workload-socket\") pod \"e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-fln48\" (UID: \"904f96ca-921a-4d2e-8245-46500d6b7d0a\") " pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-fln48" Oct 30 19:03:16.926533 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:16.926121 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/904f96ca-921a-4d2e-8245-46500d6b7d0a-credential-socket\") pod \"e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-fln48\" (UID: \"904f96ca-921a-4d2e-8245-46500d6b7d0a\") " pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-fln48" Oct 30 19:03:16.926533 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:16.926181 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/904f96ca-921a-4d2e-8245-46500d6b7d0a-istio-data\") pod \"e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-fln48\" (UID: \"904f96ca-921a-4d2e-8245-46500d6b7d0a\") " pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-fln48" Oct 30 19:03:16.926533 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:16.926312 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/904f96ca-921a-4d2e-8245-46500d6b7d0a-workload-certs\") pod \"e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-fln48\" (UID: \"904f96ca-921a-4d2e-8245-46500d6b7d0a\") " pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-fln48" Oct 30 19:03:16.926533 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:16.926357 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/904f96ca-921a-4d2e-8245-46500d6b7d0a-workload-socket\") pod \"e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-fln48\" (UID: \"904f96ca-921a-4d2e-8245-46500d6b7d0a\") " pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-fln48" Oct 30 19:03:16.926533 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:16.926383 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/904f96ca-921a-4d2e-8245-46500d6b7d0a-istiod-ca-cert\") pod \"e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-fln48\" (UID: \"904f96ca-921a-4d2e-8245-46500d6b7d0a\") " pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-fln48" Oct 30 19:03:16.926533 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:16.926486 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/904f96ca-921a-4d2e-8245-46500d6b7d0a-credential-socket\") pod \"e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-fln48\" (UID: \"904f96ca-921a-4d2e-8245-46500d6b7d0a\") " pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-fln48" Oct 30 19:03:16.927125 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:16.926721 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/904f96ca-921a-4d2e-8245-46500d6b7d0a-istio-data\") pod \"e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-fln48\" (UID: \"904f96ca-921a-4d2e-8245-46500d6b7d0a\") " pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-fln48" Oct 30 19:03:16.928248 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:16.928225 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/904f96ca-921a-4d2e-8245-46500d6b7d0a-istio-envoy\") pod \"e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-fln48\" (UID: \"904f96ca-921a-4d2e-8245-46500d6b7d0a\") " pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-fln48" Oct 30 19:03:16.928485 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:16.928468 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/904f96ca-921a-4d2e-8245-46500d6b7d0a-istio-podinfo\") pod \"e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-fln48\" (UID: \"904f96ca-921a-4d2e-8245-46500d6b7d0a\") " pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-fln48" Oct 30 19:03:16.961499 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:16.961436 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/904f96ca-921a-4d2e-8245-46500d6b7d0a-istio-token\") pod \"e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-fln48\" (UID: \"904f96ca-921a-4d2e-8245-46500d6b7d0a\") " pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-fln48" Oct 30 19:03:16.961696 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:16.961678 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txjt7\" (UniqueName: \"kubernetes.io/projected/904f96ca-921a-4d2e-8245-46500d6b7d0a-kube-api-access-txjt7\") pod \"e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-fln48\" (UID: \"904f96ca-921a-4d2e-8245-46500d6b7d0a\") " pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-fln48" Oct 30 19:03:17.017623 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:17.017590 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-fln48" Oct 30 19:03:17.348744 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:17.348595 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-fln48"] Oct 30 19:03:17.352833 ip-10-0-140-163 kubenswrapper[6960]: W1030 19:03:17.352779 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod904f96ca_921a_4d2e_8245_46500d6b7d0a.slice/crio-ff244b47abee055e9f802b2bb6d5dbf2b78afbcf27e19e27743d91ff4d863ce4 WatchSource:0}: Error finding container ff244b47abee055e9f802b2bb6d5dbf2b78afbcf27e19e27743d91ff4d863ce4: Status 404 returned error can't find the container with id ff244b47abee055e9f802b2bb6d5dbf2b78afbcf27e19e27743d91ff4d863ce4 Oct 30 19:03:17.355606 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:17.355569 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 19:03:17.355693 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:17.355642 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 19:03:17.355693 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:17.355672 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 19:03:17.654094 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:17.653997 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:18.042070 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:18.041979 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-fln48" event={"ID":"904f96ca-921a-4d2e-8245-46500d6b7d0a","Type":"ContainerStarted","Data":"9a77e88de0d86c0fb40f91c3e22c86a41c5d36c6642a1b4deb273aa333fc1250"} Oct 30 19:03:18.042070 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:18.042040 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-fln48" event={"ID":"904f96ca-921a-4d2e-8245-46500d6b7d0a","Type":"ContainerStarted","Data":"ff244b47abee055e9f802b2bb6d5dbf2b78afbcf27e19e27743d91ff4d863ce4"} Oct 30 19:03:18.077747 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:18.074102 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-fln48" podStartSLOduration=2.074042834 podStartE2EDuration="2.074042834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:03:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:03:18.071244368 +0000 UTC m=+1765.600275509" watchObservedRunningTime="2026-10-30 19:03:18.074042834 +0000 UTC m=+1765.603073974" Oct 30 19:03:18.679168 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:18.673620 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:19.018180 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:19.018069 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-fln48" Oct 30 19:03:19.020046 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:19.020020 6960 patch_prober.go:29] interesting pod/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-fln48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Oct 30 19:03:19.020196 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:19.020065 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-fln48" podUID="904f96ca-921a-4d2e-8245-46500d6b7d0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Oct 30 19:03:19.720929 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:19.720888 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:19.987311 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:19.987235 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b0e009-bfd774574-dl2n4" Oct 30 19:03:19.987534 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:19.987515 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b0e009-bb65989f9-m7q2f" Oct 30 19:03:20.018270 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:20.018240 6960 patch_prober.go:29] interesting pod/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-fln48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Oct 30 19:03:20.018429 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:20.018289 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-fln48" podUID="904f96ca-921a-4d2e-8245-46500d6b7d0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Oct 30 19:03:20.913018 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:20.912984 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:21.018311 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:21.018275 6960 patch_prober.go:29] interesting pod/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-fln48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Oct 30 19:03:21.018486 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:21.018336 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-fln48" podUID="904f96ca-921a-4d2e-8245-46500d6b7d0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Oct 30 19:03:22.018781 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:22.018750 6960 patch_prober.go:29] interesting pod/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-fln48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Oct 30 19:03:22.019202 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:22.018805 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-fln48" podUID="904f96ca-921a-4d2e-8245-46500d6b7d0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Oct 30 19:03:22.148948 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:22.148908 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:23.018099 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:23.018067 6960 patch_prober.go:29] interesting pod/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-fln48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Oct 30 19:03:23.018321 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:23.018117 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-fln48" podUID="904f96ca-921a-4d2e-8245-46500d6b7d0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Oct 30 19:03:23.120826 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:23.120794 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b0e009-5c7656bc96-lqqdw"] Oct 30 19:03:23.125760 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:23.125739 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b0e009-5c7656bc96-lqqdw" Oct 30 19:03:23.143068 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:23.143042 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-b0e009-serving-cert\"" Oct 30 19:03:23.147415 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:23.147384 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b0e009-5c7656bc96-lqqdw"] Oct 30 19:03:23.179693 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:23.179104 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snhn5\" (UniqueName: \"kubernetes.io/projected/696a841d-b511-4430-a236-d61ff1832626-kube-api-access-snhn5\") pod \"maas-api-e2e-shared-b-w2-b0e009-5c7656bc96-lqqdw\" (UID: \"696a841d-b511-4430-a236-d61ff1832626\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b0e009-5c7656bc96-lqqdw" Oct 30 19:03:23.179693 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:23.179234 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/696a841d-b511-4430-a236-d61ff1832626-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-b0e009-5c7656bc96-lqqdw\" (UID: \"696a841d-b511-4430-a236-d61ff1832626\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b0e009-5c7656bc96-lqqdw" Oct 30 19:03:23.179693 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:23.179429 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/696a841d-b511-4430-a236-d61ff1832626-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-b0e009-5c7656bc96-lqqdw\" (UID: \"696a841d-b511-4430-a236-d61ff1832626\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b0e009-5c7656bc96-lqqdw" Oct 30 19:03:23.179693 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:23.179496 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/696a841d-b511-4430-a236-d61ff1832626-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-b0e009-5c7656bc96-lqqdw\" (UID: \"696a841d-b511-4430-a236-d61ff1832626\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b0e009-5c7656bc96-lqqdw" Oct 30 19:03:23.225610 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:23.225582 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:23.294109 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:23.285783 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/696a841d-b511-4430-a236-d61ff1832626-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-b0e009-5c7656bc96-lqqdw\" (UID: \"696a841d-b511-4430-a236-d61ff1832626\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b0e009-5c7656bc96-lqqdw" Oct 30 19:03:23.294109 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:23.285882 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/696a841d-b511-4430-a236-d61ff1832626-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-b0e009-5c7656bc96-lqqdw\" (UID: \"696a841d-b511-4430-a236-d61ff1832626\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b0e009-5c7656bc96-lqqdw" Oct 30 19:03:23.294109 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:23.285983 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snhn5\" (UniqueName: \"kubernetes.io/projected/696a841d-b511-4430-a236-d61ff1832626-kube-api-access-snhn5\") pod \"maas-api-e2e-shared-b-w2-b0e009-5c7656bc96-lqqdw\" (UID: \"696a841d-b511-4430-a236-d61ff1832626\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b0e009-5c7656bc96-lqqdw" Oct 30 19:03:23.294109 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:23.286077 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/696a841d-b511-4430-a236-d61ff1832626-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-b0e009-5c7656bc96-lqqdw\" (UID: \"696a841d-b511-4430-a236-d61ff1832626\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b0e009-5c7656bc96-lqqdw" Oct 30 19:03:23.294109 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:03:23.286413 6960 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b0e009-serving-cert: secret "maas-api-e2e-shared-b-w2-b0e009-serving-cert" not found Oct 30 19:03:23.294109 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:03:23.286483 6960 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/696a841d-b511-4430-a236-d61ff1832626-maas-api-tls podName:696a841d-b511-4430-a236-d61ff1832626 nodeName:}" failed. No retries permitted until 2026-10-30 19:03:23.786458899 +0000 UTC m=+1771.315490021 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/696a841d-b511-4430-a236-d61ff1832626-maas-api-tls") pod "maas-api-e2e-shared-b-w2-b0e009-5c7656bc96-lqqdw" (UID: "696a841d-b511-4430-a236-d61ff1832626") : secret "maas-api-e2e-shared-b-w2-b0e009-serving-cert" not found Oct 30 19:03:23.304492 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:23.295186 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/696a841d-b511-4430-a236-d61ff1832626-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-b0e009-5c7656bc96-lqqdw\" (UID: \"696a841d-b511-4430-a236-d61ff1832626\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b0e009-5c7656bc96-lqqdw" Oct 30 19:03:23.328076 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:23.328043 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/696a841d-b511-4430-a236-d61ff1832626-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-b0e009-5c7656bc96-lqqdw\" (UID: \"696a841d-b511-4430-a236-d61ff1832626\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b0e009-5c7656bc96-lqqdw" Oct 30 19:03:23.334184 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:23.333898 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snhn5\" (UniqueName: \"kubernetes.io/projected/696a841d-b511-4430-a236-d61ff1832626-kube-api-access-snhn5\") pod \"maas-api-e2e-shared-b-w2-b0e009-5c7656bc96-lqqdw\" (UID: \"696a841d-b511-4430-a236-d61ff1832626\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b0e009-5c7656bc96-lqqdw" Oct 30 19:03:23.380553 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:23.380473 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:23.676322 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:23.676268 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:23.793715 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:23.793679 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/696a841d-b511-4430-a236-d61ff1832626-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-b0e009-5c7656bc96-lqqdw\" (UID: \"696a841d-b511-4430-a236-d61ff1832626\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b0e009-5c7656bc96-lqqdw" Oct 30 19:03:23.796604 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:23.796578 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/696a841d-b511-4430-a236-d61ff1832626-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-b0e009-5c7656bc96-lqqdw\" (UID: \"696a841d-b511-4430-a236-d61ff1832626\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b0e009-5c7656bc96-lqqdw" Oct 30 19:03:24.009224 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:03:24.007946 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 19:03:24.026171 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:24.024549 6960 patch_prober.go:29] interesting pod/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-fln48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Oct 30 19:03:24.026171 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:24.024610 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-fln48" podUID="904f96ca-921a-4d2e-8245-46500d6b7d0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Oct 30 19:03:24.049600 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:24.049558 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b0e009-5c7656bc96-lqqdw" Oct 30 19:03:24.372923 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:24.372892 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b0e009-5c7656bc96-lqqdw"] Oct 30 19:03:24.374075 ip-10-0-140-163 kubenswrapper[6960]: W1030 19:03:24.374047 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod696a841d_b511_4430_a236_d61ff1832626.slice/crio-1390c906816f8a943a45eb7f5e00700489591831b56732f12acc6d200333a387 WatchSource:0}: Error finding container 1390c906816f8a943a45eb7f5e00700489591831b56732f12acc6d200333a387: Status 404 returned error can't find the container with id 1390c906816f8a943a45eb7f5e00700489591831b56732f12acc6d200333a387 Oct 30 19:03:24.470308 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:24.470271 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:25.023135 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:25.023109 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-fln48" Oct 30 19:03:25.023409 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:25.023391 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-fln48" Oct 30 19:03:25.024611 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:25.024587 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-fln48" Oct 30 19:03:25.132230 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:25.132195 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b0e009-5c7656bc96-lqqdw" event={"ID":"696a841d-b511-4430-a236-d61ff1832626","Type":"ContainerStarted","Data":"976cdf277796caa6e7cc51d599a4e19494c24270be5d1306546bf69c4a967112"} Oct 30 19:03:25.132431 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:25.132257 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b0e009-5c7656bc96-lqqdw" event={"ID":"696a841d-b511-4430-a236-d61ff1832626","Type":"ContainerStarted","Data":"1390c906816f8a943a45eb7f5e00700489591831b56732f12acc6d200333a387"} Oct 30 19:03:25.132431 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:25.132415 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b0e009-5c7656bc96-lqqdw" Oct 30 19:03:25.165865 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:25.165821 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b0e009-5c7656bc96-lqqdw" podStartSLOduration=1.742411616 podStartE2EDuration="2.165807652s" totalImagesPullingTime="423.396036ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:03:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:03:25.161952871 +0000 UTC m=+1772.690984008" watchObservedRunningTime="2026-10-30 19:03:25.165807652 +0000 UTC m=+1772.694838790" Oct 30 19:03:25.734462 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:25.734421 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:26.953912 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:26.953876 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:27.853619 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:27.852712 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:29.002532 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:29.002496 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:29.862669 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:29.862632 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:30.774169 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:30.773461 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:31.144279 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:31.144247 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b0e009-5c7656bc96-lqqdw" Oct 30 19:03:31.806420 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:31.806384 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:32.990386 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:32.990320 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:34.237739 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:34.237700 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:35.110188 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:35.110133 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:36.130347 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:36.130292 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:36.976462 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:36.976430 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:38.318775 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:38.318743 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:39.318909 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:39.318868 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:40.276921 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:40.276889 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:41.117045 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:41.117005 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:42.271655 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:42.271567 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:42.503372 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:42.499927 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:43.518582 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:43.518551 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:44.428655 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:44.428619 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:44.820460 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:44.820417 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:45.526724 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:45.526685 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:45.658976 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:45.658928 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:45.818983 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:45.818938 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:46.751038 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:46.751003 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:47.785653 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:47.784863 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:48.806819 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:48.806786 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:49.414941 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:49.414903 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:49.551080 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:49.551049 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:49.872077 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:49.872043 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:50.772175 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:50.772118 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:50.970774 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:50.970087 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:51.129095 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:51.129061 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:51.977569 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:51.977534 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:52.713231 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:52.713197 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:52.970913 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:52.970824 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:53.896518 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:53.896486 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:53.942879 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:03:53.942839 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 19:03:54.782188 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:54.782122 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:55.630966 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:55.630932 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:56.423771 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:56.423717 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:56.843854 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:56.843807 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:56.952480 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:56.952448 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:57.218499 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:57.218414 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:58.017061 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:58.017030 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:58.827191 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:58.827124 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:03:59.699057 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:03:59.699017 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:00.578584 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:00.578553 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:01.290088 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:01.290050 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:02.093349 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:02.093312 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:02.920774 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:02.920732 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:03.031238 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:03.031201 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:03.172811 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:03.172695 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:04.136524 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:04.136486 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:04.937381 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:04.937350 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:05.523204 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:05.510774 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:06.108350 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:06.108311 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:07.068247 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:07.068131 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:07.922098 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:07.922065 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:08.783862 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:08.783822 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:09.671961 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:09.671928 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:09.824964 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:09.824910 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:10.015026 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:10.014936 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7417-12354c"] Oct 30 19:04:10.020673 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:10.020649 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7417-12354c" Oct 30 19:04:10.025191 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:10.025123 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-7h66b\"" Oct 30 19:04:10.027247 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:10.026981 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7417-12354c"] Oct 30 19:04:10.091491 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:10.091455 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gj8xl\" (UniqueName: \"kubernetes.io/projected/42ed667a-9551-4536-885d-378474bb8cc3-kube-api-access-gj8xl\") pod \"test-curl-7417-12354c\" (UID: \"42ed667a-9551-4536-885d-378474bb8cc3\") " pod="opendatahub/test-curl-7417-12354c" Oct 30 19:04:10.193011 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:10.192979 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj8xl\" (UniqueName: \"kubernetes.io/projected/42ed667a-9551-4536-885d-378474bb8cc3-kube-api-access-gj8xl\") pod \"test-curl-7417-12354c\" (UID: \"42ed667a-9551-4536-885d-378474bb8cc3\") " pod="opendatahub/test-curl-7417-12354c" Oct 30 19:04:10.207435 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:10.207409 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj8xl\" (UniqueName: \"kubernetes.io/projected/42ed667a-9551-4536-885d-378474bb8cc3-kube-api-access-gj8xl\") pod \"test-curl-7417-12354c\" (UID: \"42ed667a-9551-4536-885d-378474bb8cc3\") " pod="opendatahub/test-curl-7417-12354c" Oct 30 19:04:10.339796 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:10.339763 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7417-12354c" Oct 30 19:04:10.625395 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:10.625359 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7417-12354c"] Oct 30 19:04:10.626682 ip-10-0-140-163 kubenswrapper[6960]: W1030 19:04:10.626653 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod42ed667a_9551_4536_885d_378474bb8cc3.slice/crio-5927eece227293ecc8c0fa9d667cc883432e01662ecdee1634048aeb78ac0e72 WatchSource:0}: Error finding container 5927eece227293ecc8c0fa9d667cc883432e01662ecdee1634048aeb78ac0e72: Status 404 returned error can't find the container with id 5927eece227293ecc8c0fa9d667cc883432e01662ecdee1634048aeb78ac0e72 Oct 30 19:04:10.696617 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:10.696571 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:11.555221 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:11.555129 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7417-12354c" event={"ID":"42ed667a-9551-4536-885d-378474bb8cc3","Type":"ContainerStarted","Data":"e1ec49f0c942c801b73a16170ff1e8a3114b901ebb06d31cfdb32fa6c7443b99"} Oct 30 19:04:11.555221 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:11.555217 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7417-12354c" event={"ID":"42ed667a-9551-4536-885d-378474bb8cc3","Type":"ContainerStarted","Data":"5927eece227293ecc8c0fa9d667cc883432e01662ecdee1634048aeb78ac0e72"} Oct 30 19:04:11.618139 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:11.618084 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7417-12354c" podStartSLOduration=2.618070799 podStartE2EDuration="2.618070799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:04:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:04:11.615729459 +0000 UTC m=+1819.144760600" watchObservedRunningTime="2026-10-30 19:04:11.618070799 +0000 UTC m=+1819.147101937" Oct 30 19:04:11.779649 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:11.779601 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:12.374984 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:12.374950 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7417-12354c"] Oct 30 19:04:12.379867 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:12.379835 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7417-12354c"] Oct 30 19:04:12.540534 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:12.540501 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7417-cd2ae6"] Oct 30 19:04:12.541301 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:12.541282 6960 cpu_manager.go:414] "containerMap: removing container" podUID="42ed667a-9551-4536-885d-378474bb8cc3" containerName="test-curl-7417-12354c" Oct 30 19:04:12.541364 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:12.541302 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42ed667a-9551-4536-885d-378474bb8cc3" containerName="test-curl-7417-12354c" Oct 30 19:04:12.541445 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:12.541436 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="42ed667a-9551-4536-885d-378474bb8cc3" containerName="test-curl-7417-12354c" Oct 30 19:04:12.544664 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:12.544647 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7417-cd2ae6" Oct 30 19:04:12.554005 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:12.553976 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7417-cd2ae6"] Oct 30 19:04:12.625242 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:12.625163 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:12.636135 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:12.636106 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9nkz\" (UniqueName: \"kubernetes.io/projected/12e361f2-1bea-43e8-aad1-38e7227e67b2-kube-api-access-v9nkz\") pod \"test-curl-7417-cd2ae6\" (UID: \"12e361f2-1bea-43e8-aad1-38e7227e67b2\") " pod="opendatahub/test-curl-7417-cd2ae6" Oct 30 19:04:12.738417 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:12.738376 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9nkz\" (UniqueName: \"kubernetes.io/projected/12e361f2-1bea-43e8-aad1-38e7227e67b2-kube-api-access-v9nkz\") pod \"test-curl-7417-cd2ae6\" (UID: \"12e361f2-1bea-43e8-aad1-38e7227e67b2\") " pod="opendatahub/test-curl-7417-cd2ae6" Oct 30 19:04:12.750809 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:12.750780 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9nkz\" (UniqueName: \"kubernetes.io/projected/12e361f2-1bea-43e8-aad1-38e7227e67b2-kube-api-access-v9nkz\") pod \"test-curl-7417-cd2ae6\" (UID: \"12e361f2-1bea-43e8-aad1-38e7227e67b2\") " pod="opendatahub/test-curl-7417-cd2ae6" Oct 30 19:04:12.871354 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:12.871321 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7417-cd2ae6" Oct 30 19:04:13.204495 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:13.200733 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7417-cd2ae6"] Oct 30 19:04:13.211764 ip-10-0-140-163 kubenswrapper[6960]: W1030 19:04:13.211705 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod12e361f2_1bea_43e8_aad1_38e7227e67b2.slice/crio-f46f800c8d4fe11f33ffe8403be25b80b579da7bd3a6f957eb6c4f15e88d08b7 WatchSource:0}: Error finding container f46f800c8d4fe11f33ffe8403be25b80b579da7bd3a6f957eb6c4f15e88d08b7: Status 404 returned error can't find the container with id f46f800c8d4fe11f33ffe8403be25b80b579da7bd3a6f957eb6c4f15e88d08b7 Oct 30 19:04:13.601167 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:13.600597 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:13.612408 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:13.606262 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7417-12354c" podUID="42ed667a-9551-4536-885d-378474bb8cc3" containerName="test-curl-7417-12354c" containerID="cri-o://e1ec49f0c942c801b73a16170ff1e8a3114b901ebb06d31cfdb32fa6c7443b99" gracePeriod=2 Oct 30 19:04:13.612408 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:13.607448 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7417-cd2ae6" event={"ID":"12e361f2-1bea-43e8-aad1-38e7227e67b2","Type":"ContainerStarted","Data":"1b7e70f3468407a2cdcb146bcd56c06660e7eacc5a88e108fd7682758ed81a4b"} Oct 30 19:04:13.612408 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:13.607492 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7417-cd2ae6" event={"ID":"12e361f2-1bea-43e8-aad1-38e7227e67b2","Type":"ContainerStarted","Data":"f46f800c8d4fe11f33ffe8403be25b80b579da7bd3a6f957eb6c4f15e88d08b7"} Oct 30 19:04:13.634746 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:13.634684 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7417-cd2ae6" podStartSLOduration=1.6346660640000001 podStartE2EDuration="1.634666064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:04:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:04:13.628203833 +0000 UTC m=+1821.157234970" watchObservedRunningTime="2026-10-30 19:04:13.634666064 +0000 UTC m=+1821.163697203" Oct 30 19:04:14.534549 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:14.534511 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7417-cd2ae6"] Oct 30 19:04:14.543086 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:14.543054 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7417-cd2ae6"] Oct 30 19:04:14.700296 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:14.700248 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:14.702658 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:14.702639 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7417-2c674a"] Oct 30 19:04:14.703498 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:14.703472 6960 cpu_manager.go:414] "containerMap: removing container" podUID="12e361f2-1bea-43e8-aad1-38e7227e67b2" containerName="test-curl-7417-cd2ae6" Oct 30 19:04:14.703498 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:14.703494 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12e361f2-1bea-43e8-aad1-38e7227e67b2" containerName="test-curl-7417-cd2ae6" Oct 30 19:04:14.703715 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:14.703704 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="12e361f2-1bea-43e8-aad1-38e7227e67b2" containerName="test-curl-7417-cd2ae6" Oct 30 19:04:14.707365 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:14.707341 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7417-2c674a" Oct 30 19:04:14.711995 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:14.711968 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7417-2c674a"] Oct 30 19:04:14.884852 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:14.884816 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grfmq\" (UniqueName: \"kubernetes.io/projected/338442ab-4ca1-456a-8309-d98c65f08611-kube-api-access-grfmq\") pod \"test-curl-7417-2c674a\" (UID: \"338442ab-4ca1-456a-8309-d98c65f08611\") " pod="opendatahub/test-curl-7417-2c674a" Oct 30 19:04:14.986720 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:14.986690 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grfmq\" (UniqueName: \"kubernetes.io/projected/338442ab-4ca1-456a-8309-d98c65f08611-kube-api-access-grfmq\") pod \"test-curl-7417-2c674a\" (UID: \"338442ab-4ca1-456a-8309-d98c65f08611\") " pod="opendatahub/test-curl-7417-2c674a" Oct 30 19:04:14.999893 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:14.999861 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grfmq\" (UniqueName: \"kubernetes.io/projected/338442ab-4ca1-456a-8309-d98c65f08611-kube-api-access-grfmq\") pod \"test-curl-7417-2c674a\" (UID: \"338442ab-4ca1-456a-8309-d98c65f08611\") " pod="opendatahub/test-curl-7417-2c674a" Oct 30 19:04:15.039637 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:15.039605 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7417-2c674a" Oct 30 19:04:15.374755 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:15.374722 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7417-2c674a"] Oct 30 19:04:15.376783 ip-10-0-140-163 kubenswrapper[6960]: W1030 19:04:15.376758 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod338442ab_4ca1_456a_8309_d98c65f08611.slice/crio-3e6f2e1024de6b251879b1c1560f9df261c7baecb3a7810e683a1f5b3084e16c WatchSource:0}: Error finding container 3e6f2e1024de6b251879b1c1560f9df261c7baecb3a7810e683a1f5b3084e16c: Status 404 returned error can't find the container with id 3e6f2e1024de6b251879b1c1560f9df261c7baecb3a7810e683a1f5b3084e16c Oct 30 19:04:15.616954 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:15.616911 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:15.623418 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:15.623355 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7417-2c674a" event={"ID":"338442ab-4ca1-456a-8309-d98c65f08611","Type":"ContainerStarted","Data":"a6c56316891f8707d5647433ad32d1214105a5072b4dae8331c7ba717d42e42e"} Oct 30 19:04:15.623418 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:15.623402 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7417-2c674a" event={"ID":"338442ab-4ca1-456a-8309-d98c65f08611","Type":"ContainerStarted","Data":"3e6f2e1024de6b251879b1c1560f9df261c7baecb3a7810e683a1f5b3084e16c"} Oct 30 19:04:15.623584 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:15.623493 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7417-cd2ae6" podUID="12e361f2-1bea-43e8-aad1-38e7227e67b2" containerName="test-curl-7417-cd2ae6" containerID="cri-o://1b7e70f3468407a2cdcb146bcd56c06660e7eacc5a88e108fd7682758ed81a4b" gracePeriod=2 Oct 30 19:04:15.645105 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:15.643501 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7417-2c674a" podStartSLOduration=1.6434812619999999 podStartE2EDuration="1.643481262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:04:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:04:15.639257667 +0000 UTC m=+1823.168288808" watchObservedRunningTime="2026-10-30 19:04:15.643481262 +0000 UTC m=+1823.172512402" Oct 30 19:04:15.867312 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:15.867276 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7417-12354c" Oct 30 19:04:16.003851 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:16.003756 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42ed667a-9551-4536-885d-378474bb8cc3-kube-api-access-gj8xl\" (UniqueName: \"kubernetes.io/projected/42ed667a-9551-4536-885d-378474bb8cc3-kube-api-access-gj8xl\") pod \"42ed667a-9551-4536-885d-378474bb8cc3\" (UID: \"42ed667a-9551-4536-885d-378474bb8cc3\") " Oct 30 19:04:16.007870 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:16.007836 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42ed667a-9551-4536-885d-378474bb8cc3-kube-api-access-gj8xl" pod "42ed667a-9551-4536-885d-378474bb8cc3" (UID: "42ed667a-9551-4536-885d-378474bb8cc3"). InnerVolumeSpecName "kube-api-access-gj8xl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:04:16.106161 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:16.106113 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gj8xl\" (UniqueName: \"kubernetes.io/projected/42ed667a-9551-4536-885d-378474bb8cc3-kube-api-access-gj8xl\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:04:16.365047 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:16.365004 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7417-2c674a"] Oct 30 19:04:16.369203 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:16.369173 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7417-2c674a"] Oct 30 19:04:16.534705 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:16.534675 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7417-815731"] Oct 30 19:04:16.535582 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:16.535561 6960 cpu_manager.go:414] "containerMap: removing container" podUID="338442ab-4ca1-456a-8309-d98c65f08611" containerName="test-curl-7417-2c674a" Oct 30 19:04:16.535582 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:16.535581 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="338442ab-4ca1-456a-8309-d98c65f08611" containerName="test-curl-7417-2c674a" Oct 30 19:04:16.535802 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:16.535787 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="338442ab-4ca1-456a-8309-d98c65f08611" containerName="test-curl-7417-2c674a" Oct 30 19:04:16.539426 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:16.539406 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7417-815731" Oct 30 19:04:16.545871 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:16.545848 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7417-815731"] Oct 30 19:04:16.596675 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:16.596633 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42ed667a-9551-4536-885d-378474bb8cc3" path="/var/lib/kubelet/pods/42ed667a-9551-4536-885d-378474bb8cc3/volumes" Oct 30 19:04:16.605882 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:16.605846 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:16.648716 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:16.647636 6960 generic.go:356] "Generic (PLEG): container finished" podID="42ed667a-9551-4536-885d-378474bb8cc3" containerID="e1ec49f0c942c801b73a16170ff1e8a3114b901ebb06d31cfdb32fa6c7443b99" exitCode=137 Oct 30 19:04:16.648716 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:16.648190 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7417-12354c" Oct 30 19:04:16.648716 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:16.648263 6960 scope.go:122] "RemoveContainer" containerID="e1ec49f0c942c801b73a16170ff1e8a3114b901ebb06d31cfdb32fa6c7443b99" Oct 30 19:04:16.664095 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:16.664070 6960 scope.go:122] "RemoveContainer" containerID="e1ec49f0c942c801b73a16170ff1e8a3114b901ebb06d31cfdb32fa6c7443b99" Oct 30 19:04:16.665584 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:04:16.665541 6960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1ec49f0c942c801b73a16170ff1e8a3114b901ebb06d31cfdb32fa6c7443b99\": container with ID starting with e1ec49f0c942c801b73a16170ff1e8a3114b901ebb06d31cfdb32fa6c7443b99 not found: ID does not exist" containerID="e1ec49f0c942c801b73a16170ff1e8a3114b901ebb06d31cfdb32fa6c7443b99" Oct 30 19:04:16.665695 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:16.665595 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1ec49f0c942c801b73a16170ff1e8a3114b901ebb06d31cfdb32fa6c7443b99"} err="failed to get container status \"e1ec49f0c942c801b73a16170ff1e8a3114b901ebb06d31cfdb32fa6c7443b99\": rpc error: code = NotFound desc = could not find container \"e1ec49f0c942c801b73a16170ff1e8a3114b901ebb06d31cfdb32fa6c7443b99\": container with ID starting with e1ec49f0c942c801b73a16170ff1e8a3114b901ebb06d31cfdb32fa6c7443b99 not found: ID does not exist" Oct 30 19:04:16.714780 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:16.714738 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hz5fv\" (UniqueName: \"kubernetes.io/projected/1b410ab7-ec4d-454b-8c95-857fe4e63816-kube-api-access-hz5fv\") pod \"test-curl-7417-815731\" (UID: \"1b410ab7-ec4d-454b-8c95-857fe4e63816\") " pod="opendatahub/test-curl-7417-815731" Oct 30 19:04:16.816642 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:16.816612 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz5fv\" (UniqueName: \"kubernetes.io/projected/1b410ab7-ec4d-454b-8c95-857fe4e63816-kube-api-access-hz5fv\") pod \"test-curl-7417-815731\" (UID: \"1b410ab7-ec4d-454b-8c95-857fe4e63816\") " pod="opendatahub/test-curl-7417-815731" Oct 30 19:04:16.831586 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:16.831558 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hz5fv\" (UniqueName: \"kubernetes.io/projected/1b410ab7-ec4d-454b-8c95-857fe4e63816-kube-api-access-hz5fv\") pod \"test-curl-7417-815731\" (UID: \"1b410ab7-ec4d-454b-8c95-857fe4e63816\") " pod="opendatahub/test-curl-7417-815731" Oct 30 19:04:16.855679 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:16.855655 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7417-815731" Oct 30 19:04:17.171402 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:17.171375 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7417-815731"] Oct 30 19:04:17.173113 ip-10-0-140-163 kubenswrapper[6960]: W1030 19:04:17.173086 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1b410ab7_ec4d_454b_8c95_857fe4e63816.slice/crio-5792ea8efe90a756c5ef837eaa0726f0fa842176a58934a2c762bb7499351e12 WatchSource:0}: Error finding container 5792ea8efe90a756c5ef837eaa0726f0fa842176a58934a2c762bb7499351e12: Status 404 returned error can't find the container with id 5792ea8efe90a756c5ef837eaa0726f0fa842176a58934a2c762bb7499351e12 Oct 30 19:04:17.615547 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:17.615509 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:17.682180 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:17.678829 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7417-815731" event={"ID":"1b410ab7-ec4d-454b-8c95-857fe4e63816","Type":"ContainerStarted","Data":"f7ffb99827b8a2d577b64314783b0aeea9e7a50f79cdcd4118a53ee10cd33344"} Oct 30 19:04:17.682180 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:17.678893 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7417-815731" event={"ID":"1b410ab7-ec4d-454b-8c95-857fe4e63816","Type":"ContainerStarted","Data":"5792ea8efe90a756c5ef837eaa0726f0fa842176a58934a2c762bb7499351e12"} Oct 30 19:04:17.682180 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:17.681546 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7417-2c674a" podUID="338442ab-4ca1-456a-8309-d98c65f08611" containerName="test-curl-7417-2c674a" containerID="cri-o://a6c56316891f8707d5647433ad32d1214105a5072b4dae8331c7ba717d42e42e" gracePeriod=2 Oct 30 19:04:17.700041 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:17.699991 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7417-815731" podStartSLOduration=1.699972989 podStartE2EDuration="1.699972989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:04:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:04:17.693835868 +0000 UTC m=+1825.222867006" watchObservedRunningTime="2026-10-30 19:04:17.699972989 +0000 UTC m=+1825.229004129" Oct 30 19:04:17.882369 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:17.882315 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7417-cd2ae6" Oct 30 19:04:18.036785 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:18.036757 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12e361f2-1bea-43e8-aad1-38e7227e67b2-kube-api-access-v9nkz\" (UniqueName: \"kubernetes.io/projected/12e361f2-1bea-43e8-aad1-38e7227e67b2-kube-api-access-v9nkz\") pod \"12e361f2-1bea-43e8-aad1-38e7227e67b2\" (UID: \"12e361f2-1bea-43e8-aad1-38e7227e67b2\") " Oct 30 19:04:18.038877 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:18.038846 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12e361f2-1bea-43e8-aad1-38e7227e67b2-kube-api-access-v9nkz" pod "12e361f2-1bea-43e8-aad1-38e7227e67b2" (UID: "12e361f2-1bea-43e8-aad1-38e7227e67b2"). InnerVolumeSpecName "kube-api-access-v9nkz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:04:18.139075 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:18.138995 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9nkz\" (UniqueName: \"kubernetes.io/projected/12e361f2-1bea-43e8-aad1-38e7227e67b2-kube-api-access-v9nkz\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:04:18.446048 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:18.445970 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7417-815731"] Oct 30 19:04:18.450767 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:18.450742 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7417-815731"] Oct 30 19:04:18.594252 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:18.594220 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12e361f2-1bea-43e8-aad1-38e7227e67b2" path="/var/lib/kubelet/pods/12e361f2-1bea-43e8-aad1-38e7227e67b2/volumes" Oct 30 19:04:18.607105 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:18.607075 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:18.693233 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:18.693193 6960 generic.go:356] "Generic (PLEG): container finished" podID="12e361f2-1bea-43e8-aad1-38e7227e67b2" containerID="1b7e70f3468407a2cdcb146bcd56c06660e7eacc5a88e108fd7682758ed81a4b" exitCode=137 Oct 30 19:04:18.693452 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:18.693358 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7417-cd2ae6" Oct 30 19:04:18.693452 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:18.693440 6960 scope.go:122] "RemoveContainer" containerID="1b7e70f3468407a2cdcb146bcd56c06660e7eacc5a88e108fd7682758ed81a4b" Oct 30 19:04:18.709299 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:18.707963 6960 scope.go:122] "RemoveContainer" containerID="1b7e70f3468407a2cdcb146bcd56c06660e7eacc5a88e108fd7682758ed81a4b" Oct 30 19:04:18.709299 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:04:18.708502 6960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b7e70f3468407a2cdcb146bcd56c06660e7eacc5a88e108fd7682758ed81a4b\": container with ID starting with 1b7e70f3468407a2cdcb146bcd56c06660e7eacc5a88e108fd7682758ed81a4b not found: ID does not exist" containerID="1b7e70f3468407a2cdcb146bcd56c06660e7eacc5a88e108fd7682758ed81a4b" Oct 30 19:04:18.709299 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:18.708542 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b7e70f3468407a2cdcb146bcd56c06660e7eacc5a88e108fd7682758ed81a4b"} err="failed to get container status \"1b7e70f3468407a2cdcb146bcd56c06660e7eacc5a88e108fd7682758ed81a4b\": rpc error: code = NotFound desc = could not find container \"1b7e70f3468407a2cdcb146bcd56c06660e7eacc5a88e108fd7682758ed81a4b\": container with ID starting with 1b7e70f3468407a2cdcb146bcd56c06660e7eacc5a88e108fd7682758ed81a4b not found: ID does not exist" Oct 30 19:04:19.701351 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:19.701318 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7417-815731" podUID="1b410ab7-ec4d-454b-8c95-857fe4e63816" containerName="test-curl-7417-815731" containerID="cri-o://f7ffb99827b8a2d577b64314783b0aeea9e7a50f79cdcd4118a53ee10cd33344" gracePeriod=2 Oct 30 19:04:19.830381 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:19.830315 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:19.896895 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:19.896855 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7417-2c674a" Oct 30 19:04:19.992130 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:19.992046 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/338442ab-4ca1-456a-8309-d98c65f08611-kube-api-access-grfmq\" (UniqueName: \"kubernetes.io/projected/338442ab-4ca1-456a-8309-d98c65f08611-kube-api-access-grfmq\") pod \"338442ab-4ca1-456a-8309-d98c65f08611\" (UID: \"338442ab-4ca1-456a-8309-d98c65f08611\") " Oct 30 19:04:19.995106 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:19.994992 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/338442ab-4ca1-456a-8309-d98c65f08611-kube-api-access-grfmq" pod "338442ab-4ca1-456a-8309-d98c65f08611" (UID: "338442ab-4ca1-456a-8309-d98c65f08611"). InnerVolumeSpecName "kube-api-access-grfmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:04:20.093873 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:20.093838 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grfmq\" (UniqueName: \"kubernetes.io/projected/338442ab-4ca1-456a-8309-d98c65f08611-kube-api-access-grfmq\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:04:20.128368 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:20.128332 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:20.599033 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:20.598999 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="338442ab-4ca1-456a-8309-d98c65f08611" path="/var/lib/kubelet/pods/338442ab-4ca1-456a-8309-d98c65f08611/volumes" Oct 30 19:04:20.613632 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:20.613599 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:20.715938 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:20.715901 6960 generic.go:356] "Generic (PLEG): container finished" podID="338442ab-4ca1-456a-8309-d98c65f08611" containerID="a6c56316891f8707d5647433ad32d1214105a5072b4dae8331c7ba717d42e42e" exitCode=137 Oct 30 19:04:20.716380 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:20.715944 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7417-2c674a" Oct 30 19:04:20.716380 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:20.716006 6960 scope.go:122] "RemoveContainer" containerID="a6c56316891f8707d5647433ad32d1214105a5072b4dae8331c7ba717d42e42e" Oct 30 19:04:20.728384 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:20.728362 6960 scope.go:122] "RemoveContainer" containerID="a6c56316891f8707d5647433ad32d1214105a5072b4dae8331c7ba717d42e42e" Oct 30 19:04:20.729413 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:04:20.729313 6960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6c56316891f8707d5647433ad32d1214105a5072b4dae8331c7ba717d42e42e\": container with ID starting with a6c56316891f8707d5647433ad32d1214105a5072b4dae8331c7ba717d42e42e not found: ID does not exist" containerID="a6c56316891f8707d5647433ad32d1214105a5072b4dae8331c7ba717d42e42e" Oct 30 19:04:20.729413 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:20.729349 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6c56316891f8707d5647433ad32d1214105a5072b4dae8331c7ba717d42e42e"} err="failed to get container status \"a6c56316891f8707d5647433ad32d1214105a5072b4dae8331c7ba717d42e42e\": rpc error: code = NotFound desc = could not find container \"a6c56316891f8707d5647433ad32d1214105a5072b4dae8331c7ba717d42e42e\": container with ID starting with a6c56316891f8707d5647433ad32d1214105a5072b4dae8331c7ba717d42e42e not found: ID does not exist" Oct 30 19:04:20.762347 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:20.762306 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:21.001234 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:20.993096 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:21.970774 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:21.970701 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7417-815731" Oct 30 19:04:22.028446 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:22.028406 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:22.034731 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:22.034703 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b410ab7-ec4d-454b-8c95-857fe4e63816-kube-api-access-hz5fv\" (UniqueName: \"kubernetes.io/projected/1b410ab7-ec4d-454b-8c95-857fe4e63816-kube-api-access-hz5fv\") pod \"1b410ab7-ec4d-454b-8c95-857fe4e63816\" (UID: \"1b410ab7-ec4d-454b-8c95-857fe4e63816\") " Oct 30 19:04:22.037020 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:22.036989 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b410ab7-ec4d-454b-8c95-857fe4e63816-kube-api-access-hz5fv" pod "1b410ab7-ec4d-454b-8c95-857fe4e63816" (UID: "1b410ab7-ec4d-454b-8c95-857fe4e63816"). InnerVolumeSpecName "kube-api-access-hz5fv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:04:22.137380 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:22.137336 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hz5fv\" (UniqueName: \"kubernetes.io/projected/1b410ab7-ec4d-454b-8c95-857fe4e63816-kube-api-access-hz5fv\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:04:22.596252 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:22.596221 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b410ab7-ec4d-454b-8c95-857fe4e63816" path="/var/lib/kubelet/pods/1b410ab7-ec4d-454b-8c95-857fe4e63816/volumes" Oct 30 19:04:22.738970 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:22.738925 6960 generic.go:356] "Generic (PLEG): container finished" podID="1b410ab7-ec4d-454b-8c95-857fe4e63816" containerID="f7ffb99827b8a2d577b64314783b0aeea9e7a50f79cdcd4118a53ee10cd33344" exitCode=137 Oct 30 19:04:22.738970 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:22.738969 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7417-815731" Oct 30 19:04:22.739253 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:22.739018 6960 scope.go:122] "RemoveContainer" containerID="f7ffb99827b8a2d577b64314783b0aeea9e7a50f79cdcd4118a53ee10cd33344" Oct 30 19:04:22.749495 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:22.749279 6960 scope.go:122] "RemoveContainer" containerID="f7ffb99827b8a2d577b64314783b0aeea9e7a50f79cdcd4118a53ee10cd33344" Oct 30 19:04:22.749604 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:04:22.749566 6960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7ffb99827b8a2d577b64314783b0aeea9e7a50f79cdcd4118a53ee10cd33344\": container with ID starting with f7ffb99827b8a2d577b64314783b0aeea9e7a50f79cdcd4118a53ee10cd33344 not found: ID does not exist" containerID="f7ffb99827b8a2d577b64314783b0aeea9e7a50f79cdcd4118a53ee10cd33344" Oct 30 19:04:22.749662 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:22.749602 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7ffb99827b8a2d577b64314783b0aeea9e7a50f79cdcd4118a53ee10cd33344"} err="failed to get container status \"f7ffb99827b8a2d577b64314783b0aeea9e7a50f79cdcd4118a53ee10cd33344\": rpc error: code = NotFound desc = could not find container \"f7ffb99827b8a2d577b64314783b0aeea9e7a50f79cdcd4118a53ee10cd33344\": container with ID starting with f7ffb99827b8a2d577b64314783b0aeea9e7a50f79cdcd4118a53ee10cd33344 not found: ID does not exist" Oct 30 19:04:23.058367 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:23.058336 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:23.934285 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:04:23.934243 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 19:04:24.057209 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:24.057137 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:24.192667 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:24.192582 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:25.049278 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:25.049230 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:25.369690 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:25.369652 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:25.538598 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:25.538568 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:26.482015 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:26.481971 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:26.603999 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:26.603968 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:26.978997 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:26.978960 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:27.152176 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:27.152043 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:27.859631 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:27.859598 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:28.719764 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:28.719734 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:29.837278 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:29.837246 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:30.608638 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:30.608608 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:30.731723 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:30.731693 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:30.874605 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:30.874441 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:32.062762 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:32.062722 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:32.211549 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:32.211499 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:32.329475 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:32.328028 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:33.119744 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:33.119707 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:34.023088 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:34.023058 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:34.850546 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:34.850511 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:35.629404 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:35.629366 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:36.625123 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:36.625092 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:37.496876 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:37.496846 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:38.612192 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:38.612158 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:39.581718 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:39.581685 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:39.758997 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:39.756841 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:40.122012 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:40.121978 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-86c150/auth-test-model-86c150-kserve-84bf95dfd5-5qc64"] Oct 30 19:04:40.150326 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:40.150286 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:40.697454 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:40.697399 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:40.810950 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:40.810915 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:41.522447 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:41.522389 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:42.382777 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:42.382729 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:42.574881 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:42.574837 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:43.299760 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:43.299724 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:44.053842 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:44.053805 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:44.165653 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:44.165614 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:44.795463 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:44.795395 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:45.240390 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:45.240340 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:45.479811 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:45.479775 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:45.619168 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:45.615428 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:46.660115 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:46.660081 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:47.651799 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:47.651766 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-f7c6e3-78d88cd6dd-pffm6"] Oct 30 19:04:47.652795 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:47.652767 6960 cpu_manager.go:414] "containerMap: removing container" podUID="1b410ab7-ec4d-454b-8c95-857fe4e63816" containerName="test-curl-7417-815731" Oct 30 19:04:47.652795 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:47.652794 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b410ab7-ec4d-454b-8c95-857fe4e63816" containerName="test-curl-7417-815731" Oct 30 19:04:47.653097 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:47.653077 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b410ab7-ec4d-454b-8c95-857fe4e63816" containerName="test-curl-7417-815731" Oct 30 19:04:47.657647 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:47.657615 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f7c6e3-78d88cd6dd-pffm6" Oct 30 19:04:47.661636 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:47.661613 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-f7c6e3-serving-cert\"" Oct 30 19:04:47.688817 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:47.688786 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-f7c6e3-78d88cd6dd-pffm6"] Oct 30 19:04:47.746193 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:47.746135 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:47.761565 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:47.761537 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aeba7fc6-3fe2-4327-94cb-84c86ad4f1d1-maas-api-tls\") pod \"maas-api-e2e-extmod-b-f7c6e3-78d88cd6dd-pffm6\" (UID: \"aeba7fc6-3fe2-4327-94cb-84c86ad4f1d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f7c6e3-78d88cd6dd-pffm6" Oct 30 19:04:47.761739 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:47.761616 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aeba7fc6-3fe2-4327-94cb-84c86ad4f1d1-metrics-certs\") pod \"maas-api-e2e-extmod-b-f7c6e3-78d88cd6dd-pffm6\" (UID: \"aeba7fc6-3fe2-4327-94cb-84c86ad4f1d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f7c6e3-78d88cd6dd-pffm6" Oct 30 19:04:47.762379 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:47.762353 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sz7w6\" (UniqueName: \"kubernetes.io/projected/aeba7fc6-3fe2-4327-94cb-84c86ad4f1d1-kube-api-access-sz7w6\") pod \"maas-api-e2e-extmod-b-f7c6e3-78d88cd6dd-pffm6\" (UID: \"aeba7fc6-3fe2-4327-94cb-84c86ad4f1d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f7c6e3-78d88cd6dd-pffm6" Oct 30 19:04:47.762548 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:47.762527 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aeba7fc6-3fe2-4327-94cb-84c86ad4f1d1-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-f7c6e3-78d88cd6dd-pffm6\" (UID: \"aeba7fc6-3fe2-4327-94cb-84c86ad4f1d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f7c6e3-78d88cd6dd-pffm6" Oct 30 19:04:47.864037 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:47.864007 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz7w6\" (UniqueName: \"kubernetes.io/projected/aeba7fc6-3fe2-4327-94cb-84c86ad4f1d1-kube-api-access-sz7w6\") pod \"maas-api-e2e-extmod-b-f7c6e3-78d88cd6dd-pffm6\" (UID: \"aeba7fc6-3fe2-4327-94cb-84c86ad4f1d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f7c6e3-78d88cd6dd-pffm6" Oct 30 19:04:47.864247 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:47.864123 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aeba7fc6-3fe2-4327-94cb-84c86ad4f1d1-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-f7c6e3-78d88cd6dd-pffm6\" (UID: \"aeba7fc6-3fe2-4327-94cb-84c86ad4f1d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f7c6e3-78d88cd6dd-pffm6" Oct 30 19:04:47.864247 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:47.864238 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aeba7fc6-3fe2-4327-94cb-84c86ad4f1d1-maas-api-tls\") pod \"maas-api-e2e-extmod-b-f7c6e3-78d88cd6dd-pffm6\" (UID: \"aeba7fc6-3fe2-4327-94cb-84c86ad4f1d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f7c6e3-78d88cd6dd-pffm6" Oct 30 19:04:47.864386 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:47.864280 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aeba7fc6-3fe2-4327-94cb-84c86ad4f1d1-metrics-certs\") pod \"maas-api-e2e-extmod-b-f7c6e3-78d88cd6dd-pffm6\" (UID: \"aeba7fc6-3fe2-4327-94cb-84c86ad4f1d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f7c6e3-78d88cd6dd-pffm6" Oct 30 19:04:47.865598 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:47.865562 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aeba7fc6-3fe2-4327-94cb-84c86ad4f1d1-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-f7c6e3-78d88cd6dd-pffm6\" (UID: \"aeba7fc6-3fe2-4327-94cb-84c86ad4f1d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f7c6e3-78d88cd6dd-pffm6" Oct 30 19:04:47.867177 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:47.867090 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aeba7fc6-3fe2-4327-94cb-84c86ad4f1d1-maas-api-tls\") pod \"maas-api-e2e-extmod-b-f7c6e3-78d88cd6dd-pffm6\" (UID: \"aeba7fc6-3fe2-4327-94cb-84c86ad4f1d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f7c6e3-78d88cd6dd-pffm6" Oct 30 19:04:47.869371 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:47.869349 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aeba7fc6-3fe2-4327-94cb-84c86ad4f1d1-metrics-certs\") pod \"maas-api-e2e-extmod-b-f7c6e3-78d88cd6dd-pffm6\" (UID: \"aeba7fc6-3fe2-4327-94cb-84c86ad4f1d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f7c6e3-78d88cd6dd-pffm6" Oct 30 19:04:47.882798 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:47.882729 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sz7w6\" (UniqueName: \"kubernetes.io/projected/aeba7fc6-3fe2-4327-94cb-84c86ad4f1d1-kube-api-access-sz7w6\") pod \"maas-api-e2e-extmod-b-f7c6e3-78d88cd6dd-pffm6\" (UID: \"aeba7fc6-3fe2-4327-94cb-84c86ad4f1d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f7c6e3-78d88cd6dd-pffm6" Oct 30 19:04:47.894123 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:47.894030 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:47.976750 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:47.976656 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f7c6e3-78d88cd6dd-pffm6" Oct 30 19:04:48.356064 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:48.355892 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-f7c6e3-78d88cd6dd-pffm6"] Oct 30 19:04:48.377179 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:48.374798 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:48.382092 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:48.381222 6960 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:04:48.565573 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:48.565536 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:48.860187 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:48.860073 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:48.887178 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:48.883697 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-784fbf/auth-test-model-784fbf-kserve-7894bd84d9-2qwx4"] Oct 30 19:04:49.133453 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:49.114891 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:49.192026 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:49.175707 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f7c6e3-78d88cd6dd-pffm6" event={"ID":"aeba7fc6-3fe2-4327-94cb-84c86ad4f1d1","Type":"ContainerStarted","Data":"3c5ba99be827caff4becabd5ef04dbb483bddade964ec158ad39ea319262cb29"} Oct 30 19:04:49.384355 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:49.384269 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:49.668923 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:49.668824 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:50.160885 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:50.160847 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:50.185178 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:50.185122 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f7c6e3-78d88cd6dd-pffm6" event={"ID":"aeba7fc6-3fe2-4327-94cb-84c86ad4f1d1","Type":"ContainerStarted","Data":"a2802603b8acf859798b9022e561737233e434dfc14081b24892c0c6f37a59d7"} Oct 30 19:04:50.185363 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:50.185216 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f7c6e3-78d88cd6dd-pffm6" Oct 30 19:04:50.216795 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:50.216741 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f7c6e3-78d88cd6dd-pffm6" podStartSLOduration=2.514803882 podStartE2EDuration="3.216725607s" totalImagesPullingTime="701.921725ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:04:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:04:50.212212601 +0000 UTC m=+1857.741243742" watchObservedRunningTime="2026-10-30 19:04:50.216725607 +0000 UTC m=+1857.745756744" Oct 30 19:04:50.366168 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:50.360731 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:50.536555 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:50.533906 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:50.756421 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:50.755870 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:50.999007 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:50.998974 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:51.958179 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:51.948339 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:52.711994 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:52.711961 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:52.864929 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:52.864890 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:53.008289 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:53.008200 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:53.935180 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:04:53.935117 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 19:04:54.027720 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:54.027687 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:55.146206 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:55.146169 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-86c150/auth-test-model-86c150-kserve-84bf95dfd5-5qc64" podUID="7ee03adc-306c-4acb-862b-929b8433b6ac" containerName="main" containerID="cri-o://e8c27f55957af342da8da48a9a54ac64c76a3ea469a0d3adf600eb22b9237233" gracePeriod=45 Oct 30 19:04:55.373901 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:55.373825 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:55.436956 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:55.436932 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-86c150/auth-test-model-86c150-kserve-84bf95dfd5-5qc64" Oct 30 19:04:55.589288 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:55.585713 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:55.608906 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:55.598457 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ee03adc-306c-4acb-862b-929b8433b6ac-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ee03adc-306c-4acb-862b-929b8433b6ac-dshm\") pod \"7ee03adc-306c-4acb-862b-929b8433b6ac\" (UID: \"7ee03adc-306c-4acb-862b-929b8433b6ac\") " Oct 30 19:04:55.608906 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:55.598566 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ee03adc-306c-4acb-862b-929b8433b6ac-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ee03adc-306c-4acb-862b-929b8433b6ac-tmp-dir\") pod \"7ee03adc-306c-4acb-862b-929b8433b6ac\" (UID: \"7ee03adc-306c-4acb-862b-929b8433b6ac\") " Oct 30 19:04:55.608906 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:55.598668 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ee03adc-306c-4acb-862b-929b8433b6ac-kube-api-access-v8lnt\" (UniqueName: \"kubernetes.io/projected/7ee03adc-306c-4acb-862b-929b8433b6ac-kube-api-access-v8lnt\") pod \"7ee03adc-306c-4acb-862b-929b8433b6ac\" (UID: \"7ee03adc-306c-4acb-862b-929b8433b6ac\") " Oct 30 19:04:55.608906 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:55.598735 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ee03adc-306c-4acb-862b-929b8433b6ac-tls-certs\" (UniqueName: \"kubernetes.io/secret/7ee03adc-306c-4acb-862b-929b8433b6ac-tls-certs\") pod \"7ee03adc-306c-4acb-862b-929b8433b6ac\" (UID: \"7ee03adc-306c-4acb-862b-929b8433b6ac\") " Oct 30 19:04:55.608906 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:55.598833 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ee03adc-306c-4acb-862b-929b8433b6ac-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ee03adc-306c-4acb-862b-929b8433b6ac-model-cache\") pod \"7ee03adc-306c-4acb-862b-929b8433b6ac\" (UID: \"7ee03adc-306c-4acb-862b-929b8433b6ac\") " Oct 30 19:04:55.608906 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:55.598891 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ee03adc-306c-4acb-862b-929b8433b6ac-home\" (UniqueName: \"kubernetes.io/empty-dir/7ee03adc-306c-4acb-862b-929b8433b6ac-home\") pod \"7ee03adc-306c-4acb-862b-929b8433b6ac\" (UID: \"7ee03adc-306c-4acb-862b-929b8433b6ac\") " Oct 30 19:04:55.608906 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:55.600333 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ee03adc-306c-4acb-862b-929b8433b6ac-tmp-dir" pod "7ee03adc-306c-4acb-862b-929b8433b6ac" (UID: "7ee03adc-306c-4acb-862b-929b8433b6ac"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:04:55.608906 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:55.600510 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ee03adc-306c-4acb-862b-929b8433b6ac-model-cache" pod "7ee03adc-306c-4acb-862b-929b8433b6ac" (UID: "7ee03adc-306c-4acb-862b-929b8433b6ac"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:04:55.608906 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:55.600598 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ee03adc-306c-4acb-862b-929b8433b6ac-home" pod "7ee03adc-306c-4acb-862b-929b8433b6ac" (UID: "7ee03adc-306c-4acb-862b-929b8433b6ac"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:04:55.609530 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:55.609013 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ee03adc-306c-4acb-862b-929b8433b6ac-dshm" pod "7ee03adc-306c-4acb-862b-929b8433b6ac" (UID: "7ee03adc-306c-4acb-862b-929b8433b6ac"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:04:55.610330 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:55.610037 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ee03adc-306c-4acb-862b-929b8433b6ac-tls-certs" pod "7ee03adc-306c-4acb-862b-929b8433b6ac" (UID: "7ee03adc-306c-4acb-862b-929b8433b6ac"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:04:55.626396 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:55.626360 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ee03adc-306c-4acb-862b-929b8433b6ac-kube-api-access-v8lnt" pod "7ee03adc-306c-4acb-862b-929b8433b6ac" (UID: "7ee03adc-306c-4acb-862b-929b8433b6ac"). InnerVolumeSpecName "kube-api-access-v8lnt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:04:55.710257 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:55.702893 6960 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ee03adc-306c-4acb-862b-929b8433b6ac-dshm\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:04:55.710257 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:55.702928 6960 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ee03adc-306c-4acb-862b-929b8433b6ac-tmp-dir\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:04:55.710257 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:55.702942 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8lnt\" (UniqueName: \"kubernetes.io/projected/7ee03adc-306c-4acb-862b-929b8433b6ac-kube-api-access-v8lnt\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:04:55.710257 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:55.702954 6960 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ee03adc-306c-4acb-862b-929b8433b6ac-tls-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:04:55.710257 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:55.702966 6960 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ee03adc-306c-4acb-862b-929b8433b6ac-model-cache\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:04:55.710257 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:55.702977 6960 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ee03adc-306c-4acb-862b-929b8433b6ac-home\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:04:56.197690 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:56.197663 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f7c6e3-78d88cd6dd-pffm6" Oct 30 19:04:56.230069 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:56.230040 6960 generic.go:356] "Generic (PLEG): container finished" podID="7ee03adc-306c-4acb-862b-929b8433b6ac" containerID="e8c27f55957af342da8da48a9a54ac64c76a3ea469a0d3adf600eb22b9237233" exitCode=0 Oct 30 19:04:56.230264 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:56.230082 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-86c150/auth-test-model-86c150-kserve-84bf95dfd5-5qc64" Oct 30 19:04:56.230264 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:56.230130 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-86c150/auth-test-model-86c150-kserve-84bf95dfd5-5qc64" event={"ID":"7ee03adc-306c-4acb-862b-929b8433b6ac","Type":"ContainerDied","Data":"e8c27f55957af342da8da48a9a54ac64c76a3ea469a0d3adf600eb22b9237233"} Oct 30 19:04:56.230264 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:56.230195 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-86c150/auth-test-model-86c150-kserve-84bf95dfd5-5qc64" event={"ID":"7ee03adc-306c-4acb-862b-929b8433b6ac","Type":"ContainerDied","Data":"902801b2ef3d1ff26ffe482d4cc1acf701307d2623203250456d31ba9e888c7f"} Oct 30 19:04:56.230264 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:56.230212 6960 scope.go:122] "RemoveContainer" containerID="e8c27f55957af342da8da48a9a54ac64c76a3ea469a0d3adf600eb22b9237233" Oct 30 19:04:56.241289 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:56.241264 6960 scope.go:122] "RemoveContainer" containerID="e8c27f55957af342da8da48a9a54ac64c76a3ea469a0d3adf600eb22b9237233" Oct 30 19:04:56.241586 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:04:56.241568 6960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8c27f55957af342da8da48a9a54ac64c76a3ea469a0d3adf600eb22b9237233\": container with ID starting with e8c27f55957af342da8da48a9a54ac64c76a3ea469a0d3adf600eb22b9237233 not found: ID does not exist" containerID="e8c27f55957af342da8da48a9a54ac64c76a3ea469a0d3adf600eb22b9237233" Oct 30 19:04:56.241651 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:56.241595 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8c27f55957af342da8da48a9a54ac64c76a3ea469a0d3adf600eb22b9237233"} err="failed to get container status \"e8c27f55957af342da8da48a9a54ac64c76a3ea469a0d3adf600eb22b9237233\": rpc error: code = NotFound desc = could not find container \"e8c27f55957af342da8da48a9a54ac64c76a3ea469a0d3adf600eb22b9237233\": container with ID starting with e8c27f55957af342da8da48a9a54ac64c76a3ea469a0d3adf600eb22b9237233 not found: ID does not exist" Oct 30 19:04:56.269196 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:56.269167 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-86c150/auth-test-model-86c150-kserve-84bf95dfd5-5qc64"] Oct 30 19:04:56.270418 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:56.270396 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-86c150/auth-test-model-86c150-kserve-84bf95dfd5-5qc64"] Oct 30 19:04:56.597681 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:56.597643 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ee03adc-306c-4acb-862b-929b8433b6ac" path="/var/lib/kubelet/pods/7ee03adc-306c-4acb-862b-929b8433b6ac/volumes" Oct 30 19:04:56.676951 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:56.676917 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:56.890237 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:56.885776 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:57.708949 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:57.708908 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:58.116304 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:58.115220 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:59.121311 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:59.121269 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:04:59.623666 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:04:59.623634 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:00.060543 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:00.060509 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:00.180791 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:00.180710 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:00.318230 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:00.312753 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:01.167833 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:01.167793 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:02.078997 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:02.078956 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:03.176210 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:03.176172 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:03.647178 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:03.647107 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:03.785681 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:03.785643 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:04.137200 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:04.137109 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-784fbf/auth-test-model-784fbf-kserve-7894bd84d9-2qwx4" podUID="4fd1163d-2e1c-4889-b979-6a86786ba79c" containerName="main" containerID="cri-o://eb0188677df967a7fb13138076a31bc68549995eb52a270c7ae4b33150d495b6" gracePeriod=45 Oct 30 19:05:04.332235 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:04.332184 6960 generic.go:356] "Generic (PLEG): container finished" podID="4fd1163d-2e1c-4889-b979-6a86786ba79c" containerID="eb0188677df967a7fb13138076a31bc68549995eb52a270c7ae4b33150d495b6" exitCode=0 Oct 30 19:05:04.332702 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:04.332361 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-784fbf/auth-test-model-784fbf-kserve-7894bd84d9-2qwx4" event={"ID":"4fd1163d-2e1c-4889-b979-6a86786ba79c","Type":"ContainerDied","Data":"eb0188677df967a7fb13138076a31bc68549995eb52a270c7ae4b33150d495b6"} Oct 30 19:05:04.424312 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:04.424283 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-784fbf/auth-test-model-784fbf-kserve-7894bd84d9-2qwx4" Oct 30 19:05:04.466296 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:04.466239 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:04.510424 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:04.509411 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fd1163d-2e1c-4889-b979-6a86786ba79c-home\" (UniqueName: \"kubernetes.io/empty-dir/4fd1163d-2e1c-4889-b979-6a86786ba79c-home\") pod \"4fd1163d-2e1c-4889-b979-6a86786ba79c\" (UID: \"4fd1163d-2e1c-4889-b979-6a86786ba79c\") " Oct 30 19:05:04.510424 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:04.509477 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4fd1163d-2e1c-4889-b979-6a86786ba79c-tls-certs\" (UniqueName: \"kubernetes.io/secret/4fd1163d-2e1c-4889-b979-6a86786ba79c-tls-certs\") pod \"4fd1163d-2e1c-4889-b979-6a86786ba79c\" (UID: \"4fd1163d-2e1c-4889-b979-6a86786ba79c\") " Oct 30 19:05:04.510424 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:04.509637 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fd1163d-2e1c-4889-b979-6a86786ba79c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fd1163d-2e1c-4889-b979-6a86786ba79c-tmp-dir\") pod \"4fd1163d-2e1c-4889-b979-6a86786ba79c\" (UID: \"4fd1163d-2e1c-4889-b979-6a86786ba79c\") " Oct 30 19:05:04.510424 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:04.509743 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fd1163d-2e1c-4889-b979-6a86786ba79c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fd1163d-2e1c-4889-b979-6a86786ba79c-model-cache\") pod \"4fd1163d-2e1c-4889-b979-6a86786ba79c\" (UID: \"4fd1163d-2e1c-4889-b979-6a86786ba79c\") " Oct 30 19:05:04.510424 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:04.509907 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fd1163d-2e1c-4889-b979-6a86786ba79c-kube-api-access-km62k\" (UniqueName: \"kubernetes.io/projected/4fd1163d-2e1c-4889-b979-6a86786ba79c-kube-api-access-km62k\") pod \"4fd1163d-2e1c-4889-b979-6a86786ba79c\" (UID: \"4fd1163d-2e1c-4889-b979-6a86786ba79c\") " Oct 30 19:05:04.510424 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:04.509947 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fd1163d-2e1c-4889-b979-6a86786ba79c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fd1163d-2e1c-4889-b979-6a86786ba79c-dshm\") pod \"4fd1163d-2e1c-4889-b979-6a86786ba79c\" (UID: \"4fd1163d-2e1c-4889-b979-6a86786ba79c\") " Oct 30 19:05:04.518644 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:04.518611 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-frrjm"] Oct 30 19:05:04.518822 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:04.518726 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-drlj6"] Oct 30 19:05:04.522868 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:04.518998 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-drlj6" podUID="1ae98fd5-ff16-44e3-80aa-139438c86c78" containerName="istio-proxy" containerID="cri-o://31b0e3829260bd99c17c01905cb36463411dd72de7203f1e0db0ded1f8915fdc" gracePeriod=30 Oct 30 19:05:04.522868 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:04.519303 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-frrjm" podUID="a5cb9eed-a657-4907-9765-9114f12fb473" containerName="istio-proxy" containerID="cri-o://4b5fc94da92519effb48045968ef28abe79fa131cd16f3f914cac50f186e7cc5" gracePeriod=30 Oct 30 19:05:04.522868 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:04.521250 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fd1163d-2e1c-4889-b979-6a86786ba79c-tmp-dir" pod "4fd1163d-2e1c-4889-b979-6a86786ba79c" (UID: "4fd1163d-2e1c-4889-b979-6a86786ba79c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:05:04.522868 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:04.521618 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fd1163d-2e1c-4889-b979-6a86786ba79c-home" pod "4fd1163d-2e1c-4889-b979-6a86786ba79c" (UID: "4fd1163d-2e1c-4889-b979-6a86786ba79c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:05:04.523976 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:04.523931 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fd1163d-2e1c-4889-b979-6a86786ba79c-model-cache" pod "4fd1163d-2e1c-4889-b979-6a86786ba79c" (UID: "4fd1163d-2e1c-4889-b979-6a86786ba79c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:05:04.542212 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:04.530501 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fd1163d-2e1c-4889-b979-6a86786ba79c-dshm" pod "4fd1163d-2e1c-4889-b979-6a86786ba79c" (UID: "4fd1163d-2e1c-4889-b979-6a86786ba79c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:05:04.542212 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:04.530750 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fd1163d-2e1c-4889-b979-6a86786ba79c-tls-certs" pod "4fd1163d-2e1c-4889-b979-6a86786ba79c" (UID: "4fd1163d-2e1c-4889-b979-6a86786ba79c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:05:04.552559 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:04.542827 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fd1163d-2e1c-4889-b979-6a86786ba79c-kube-api-access-km62k" pod "4fd1163d-2e1c-4889-b979-6a86786ba79c" (UID: "4fd1163d-2e1c-4889-b979-6a86786ba79c"). InnerVolumeSpecName "kube-api-access-km62k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:05:04.625334 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:04.625256 6960 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fd1163d-2e1c-4889-b979-6a86786ba79c-tmp-dir\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:05:04.625334 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:04.625283 6960 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fd1163d-2e1c-4889-b979-6a86786ba79c-model-cache\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:05:04.625334 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:04.625298 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-km62k\" (UniqueName: \"kubernetes.io/projected/4fd1163d-2e1c-4889-b979-6a86786ba79c-kube-api-access-km62k\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:05:04.625334 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:04.625315 6960 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fd1163d-2e1c-4889-b979-6a86786ba79c-dshm\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:05:04.625334 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:04.625327 6960 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fd1163d-2e1c-4889-b979-6a86786ba79c-home\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:05:04.625334 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:04.625335 6960 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fd1163d-2e1c-4889-b979-6a86786ba79c-tls-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:05:04.688029 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:04.687921 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:04.892175 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:04.892128 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:05.046586 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:05.046505 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:05.342114 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:05.342082 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-784fbf/auth-test-model-784fbf-kserve-7894bd84d9-2qwx4" event={"ID":"4fd1163d-2e1c-4889-b979-6a86786ba79c","Type":"ContainerDied","Data":"a44d4656bc2c455d2a49c80cecdbbc82f7c74fa99781940e089e1e957e901cc0"} Oct 30 19:05:05.342614 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:05.342163 6960 scope.go:122] "RemoveContainer" containerID="eb0188677df967a7fb13138076a31bc68549995eb52a270c7ae4b33150d495b6" Oct 30 19:05:05.342614 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:05.342137 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-784fbf/auth-test-model-784fbf-kserve-7894bd84d9-2qwx4" Oct 30 19:05:05.362690 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:05.362662 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-784fbf/auth-test-model-784fbf-kserve-7894bd84d9-2qwx4"] Oct 30 19:05:05.369957 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:05.367896 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-784fbf/auth-test-model-784fbf-kserve-7894bd84d9-2qwx4"] Oct 30 19:05:05.510556 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:05.510523 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:06.112395 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:06.112360 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:06.581746 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:06.581717 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:06.596517 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:06.596482 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fd1163d-2e1c-4889-b979-6a86786ba79c" path="/var/lib/kubelet/pods/4fd1163d-2e1c-4889-b979-6a86786ba79c/volumes" Oct 30 19:05:06.709188 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:06.702670 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:07.634006 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:07.633966 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:07.751050 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:07.751017 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:08.381492 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:08.381460 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:09.249803 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:09.249773 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-e8c554-6dcfccfdb5-wcdsr"] Oct 30 19:05:09.250815 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:09.250788 6960 cpu_manager.go:414] "containerMap: removing container" podUID="7ee03adc-306c-4acb-862b-929b8433b6ac" containerName="main" Oct 30 19:05:09.250815 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:09.250815 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ee03adc-306c-4acb-862b-929b8433b6ac" containerName="main" Oct 30 19:05:09.251003 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:09.250848 6960 cpu_manager.go:414] "containerMap: removing container" podUID="4fd1163d-2e1c-4889-b979-6a86786ba79c" containerName="main" Oct 30 19:05:09.251003 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:09.250863 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fd1163d-2e1c-4889-b979-6a86786ba79c" containerName="main" Oct 30 19:05:09.251110 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:09.251062 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ee03adc-306c-4acb-862b-929b8433b6ac" containerName="main" Oct 30 19:05:09.251110 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:09.251080 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fd1163d-2e1c-4889-b979-6a86786ba79c" containerName="main" Oct 30 19:05:09.256531 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:09.256509 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e8c554-6dcfccfdb5-wcdsr" Oct 30 19:05:09.264017 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:09.263984 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-e8c554-serving-cert\"" Oct 30 19:05:09.275499 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:09.275446 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-e8c554-6dcfccfdb5-wcdsr"] Oct 30 19:05:09.277773 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:09.277751 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d58l4"] Oct 30 19:05:09.277970 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:09.277944 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d58l4" podUID="42b9fddf-ca5b-4181-8819-c32996d6ff0e" containerName="istio-proxy" containerID="cri-o://4de5bb8b562e6da61eaddd7604fb2e7e810953eaee3bed6dacf67fb993737fb6" gracePeriod=30 Oct 30 19:05:09.321096 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:09.321058 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:09.342658 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:09.342625 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e8ff0f2-eaf3-434d-b877-f95964be8923-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-e8c554-6dcfccfdb5-wcdsr\" (UID: \"3e8ff0f2-eaf3-434d-b877-f95964be8923\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e8c554-6dcfccfdb5-wcdsr" Oct 30 19:05:09.342815 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:09.342797 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e8ff0f2-eaf3-434d-b877-f95964be8923-maas-api-tls\") pod \"maas-api-e2e-extmod-a-e8c554-6dcfccfdb5-wcdsr\" (UID: \"3e8ff0f2-eaf3-434d-b877-f95964be8923\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e8c554-6dcfccfdb5-wcdsr" Oct 30 19:05:09.342865 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:09.342831 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8kfr\" (UniqueName: \"kubernetes.io/projected/3e8ff0f2-eaf3-434d-b877-f95964be8923-kube-api-access-p8kfr\") pod \"maas-api-e2e-extmod-a-e8c554-6dcfccfdb5-wcdsr\" (UID: \"3e8ff0f2-eaf3-434d-b877-f95964be8923\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e8c554-6dcfccfdb5-wcdsr" Oct 30 19:05:09.342944 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:09.342927 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e8ff0f2-eaf3-434d-b877-f95964be8923-metrics-certs\") pod \"maas-api-e2e-extmod-a-e8c554-6dcfccfdb5-wcdsr\" (UID: \"3e8ff0f2-eaf3-434d-b877-f95964be8923\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e8c554-6dcfccfdb5-wcdsr" Oct 30 19:05:09.438051 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:09.437999 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:09.444773 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:09.444745 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e8ff0f2-eaf3-434d-b877-f95964be8923-metrics-certs\") pod \"maas-api-e2e-extmod-a-e8c554-6dcfccfdb5-wcdsr\" (UID: \"3e8ff0f2-eaf3-434d-b877-f95964be8923\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e8c554-6dcfccfdb5-wcdsr" Oct 30 19:05:09.444954 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:09.444909 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e8ff0f2-eaf3-434d-b877-f95964be8923-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-e8c554-6dcfccfdb5-wcdsr\" (UID: \"3e8ff0f2-eaf3-434d-b877-f95964be8923\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e8c554-6dcfccfdb5-wcdsr" Oct 30 19:05:09.445023 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:09.445009 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e8ff0f2-eaf3-434d-b877-f95964be8923-maas-api-tls\") pod \"maas-api-e2e-extmod-a-e8c554-6dcfccfdb5-wcdsr\" (UID: \"3e8ff0f2-eaf3-434d-b877-f95964be8923\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e8c554-6dcfccfdb5-wcdsr" Oct 30 19:05:09.445076 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:09.445034 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8kfr\" (UniqueName: \"kubernetes.io/projected/3e8ff0f2-eaf3-434d-b877-f95964be8923-kube-api-access-p8kfr\") pod \"maas-api-e2e-extmod-a-e8c554-6dcfccfdb5-wcdsr\" (UID: \"3e8ff0f2-eaf3-434d-b877-f95964be8923\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e8c554-6dcfccfdb5-wcdsr" Oct 30 19:05:09.445665 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:09.445636 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e8ff0f2-eaf3-434d-b877-f95964be8923-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-e8c554-6dcfccfdb5-wcdsr\" (UID: \"3e8ff0f2-eaf3-434d-b877-f95964be8923\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e8c554-6dcfccfdb5-wcdsr" Oct 30 19:05:09.463201 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:09.448223 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e8ff0f2-eaf3-434d-b877-f95964be8923-maas-api-tls\") pod \"maas-api-e2e-extmod-a-e8c554-6dcfccfdb5-wcdsr\" (UID: \"3e8ff0f2-eaf3-434d-b877-f95964be8923\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e8c554-6dcfccfdb5-wcdsr" Oct 30 19:05:09.463201 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:09.448448 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e8ff0f2-eaf3-434d-b877-f95964be8923-metrics-certs\") pod \"maas-api-e2e-extmod-a-e8c554-6dcfccfdb5-wcdsr\" (UID: \"3e8ff0f2-eaf3-434d-b877-f95964be8923\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e8c554-6dcfccfdb5-wcdsr" Oct 30 19:05:09.463201 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:09.458305 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8kfr\" (UniqueName: \"kubernetes.io/projected/3e8ff0f2-eaf3-434d-b877-f95964be8923-kube-api-access-p8kfr\") pod \"maas-api-e2e-extmod-a-e8c554-6dcfccfdb5-wcdsr\" (UID: \"3e8ff0f2-eaf3-434d-b877-f95964be8923\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e8c554-6dcfccfdb5-wcdsr" Oct 30 19:05:09.546525 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:09.546487 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:09.580484 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:09.580454 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e8c554-6dcfccfdb5-wcdsr" Oct 30 19:05:09.886670 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:09.886618 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-e8c554-6dcfccfdb5-wcdsr"] Oct 30 19:05:09.894605 ip-10-0-140-163 kubenswrapper[6960]: W1030 19:05:09.892852 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e8ff0f2_eaf3_434d_b877_f95964be8923.slice/crio-70dd633a4adfbcfc39f6f44ad2d153fddee66f59cb5f70475811b6f714aa15e0 WatchSource:0}: Error finding container 70dd633a4adfbcfc39f6f44ad2d153fddee66f59cb5f70475811b6f714aa15e0: Status 404 returned error can't find the container with id 70dd633a4adfbcfc39f6f44ad2d153fddee66f59cb5f70475811b6f714aa15e0 Oct 30 19:05:10.001170 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:10.000658 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-frrjm" Oct 30 19:05:10.061931 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:10.061806 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-drlj6" Oct 30 19:05:10.099887 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:10.099852 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a5cb9eed-a657-4907-9765-9114f12fb473-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5cb9eed-a657-4907-9765-9114f12fb473-istio-podinfo\") pod \"a5cb9eed-a657-4907-9765-9114f12fb473\" (UID: \"a5cb9eed-a657-4907-9765-9114f12fb473\") " Oct 30 19:05:10.100629 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:10.100603 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5cb9eed-a657-4907-9765-9114f12fb473-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5cb9eed-a657-4907-9765-9114f12fb473-workload-certs\") pod \"a5cb9eed-a657-4907-9765-9114f12fb473\" (UID: \"a5cb9eed-a657-4907-9765-9114f12fb473\") " Oct 30 19:05:10.100897 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:10.100877 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5cb9eed-a657-4907-9765-9114f12fb473-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5cb9eed-a657-4907-9765-9114f12fb473-istio-data\") pod \"a5cb9eed-a657-4907-9765-9114f12fb473\" (UID: \"a5cb9eed-a657-4907-9765-9114f12fb473\") " Oct 30 19:05:10.101524 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:10.101502 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5cb9eed-a657-4907-9765-9114f12fb473-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5cb9eed-a657-4907-9765-9114f12fb473-credential-socket\") pod \"a5cb9eed-a657-4907-9765-9114f12fb473\" (UID: \"a5cb9eed-a657-4907-9765-9114f12fb473\") " Oct 30 19:05:10.101761 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:10.101744 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5cb9eed-a657-4907-9765-9114f12fb473-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5cb9eed-a657-4907-9765-9114f12fb473-istio-envoy\") pod \"a5cb9eed-a657-4907-9765-9114f12fb473\" (UID: \"a5cb9eed-a657-4907-9765-9114f12fb473\") " Oct 30 19:05:10.101972 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:10.101954 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5cb9eed-a657-4907-9765-9114f12fb473-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5cb9eed-a657-4907-9765-9114f12fb473-workload-socket\") pod \"a5cb9eed-a657-4907-9765-9114f12fb473\" (UID: \"a5cb9eed-a657-4907-9765-9114f12fb473\") " Oct 30 19:05:10.102715 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:10.102693 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5cb9eed-a657-4907-9765-9114f12fb473-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5cb9eed-a657-4907-9765-9114f12fb473-istiod-ca-cert\") pod \"a5cb9eed-a657-4907-9765-9114f12fb473\" (UID: \"a5cb9eed-a657-4907-9765-9114f12fb473\") " Oct 30 19:05:10.102951 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:10.102895 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5cb9eed-a657-4907-9765-9114f12fb473-kube-api-access-5p2bm\" (UniqueName: \"kubernetes.io/projected/a5cb9eed-a657-4907-9765-9114f12fb473-kube-api-access-5p2bm\") pod \"a5cb9eed-a657-4907-9765-9114f12fb473\" (UID: \"a5cb9eed-a657-4907-9765-9114f12fb473\") " Oct 30 19:05:10.103097 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:10.103081 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5cb9eed-a657-4907-9765-9114f12fb473-istio-token\" (UniqueName: \"kubernetes.io/projected/a5cb9eed-a657-4907-9765-9114f12fb473-istio-token\") pod \"a5cb9eed-a657-4907-9765-9114f12fb473\" (UID: \"a5cb9eed-a657-4907-9765-9114f12fb473\") " Oct 30 19:05:10.105993 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:10.105626 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5cb9eed-a657-4907-9765-9114f12fb473-istiod-ca-cert" pod "a5cb9eed-a657-4907-9765-9114f12fb473" (UID: "a5cb9eed-a657-4907-9765-9114f12fb473"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:05:10.106113 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:10.105997 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5cb9eed-a657-4907-9765-9114f12fb473-workload-socket" pod "a5cb9eed-a657-4907-9765-9114f12fb473" (UID: "a5cb9eed-a657-4907-9765-9114f12fb473"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:05:10.106294 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:10.106271 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5cb9eed-a657-4907-9765-9114f12fb473-credential-socket" pod "a5cb9eed-a657-4907-9765-9114f12fb473" (UID: "a5cb9eed-a657-4907-9765-9114f12fb473"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:05:10.111772 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:10.111422 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5cb9eed-a657-4907-9765-9114f12fb473-istio-data" pod "a5cb9eed-a657-4907-9765-9114f12fb473" (UID: "a5cb9eed-a657-4907-9765-9114f12fb473"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:05:10.112408 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:10.112382 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5cb9eed-a657-4907-9765-9114f12fb473-workload-certs" pod "a5cb9eed-a657-4907-9765-9114f12fb473" (UID: "a5cb9eed-a657-4907-9765-9114f12fb473"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:05:10.121087 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:10.121054 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5cb9eed-a657-4907-9765-9114f12fb473-kube-api-access-5p2bm" pod "a5cb9eed-a657-4907-9765-9114f12fb473" (UID: "a5cb9eed-a657-4907-9765-9114f12fb473"). InnerVolumeSpecName "kube-api-access-5p2bm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:05:10.121931 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:10.121828 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5cb9eed-a657-4907-9765-9114f12fb473-istio-token" pod "a5cb9eed-a657-4907-9765-9114f12fb473" (UID: "a5cb9eed-a657-4907-9765-9114f12fb473"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:05:10.122662 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:10.122641 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5cb9eed-a657-4907-9765-9114f12fb473-istio-envoy" pod "a5cb9eed-a657-4907-9765-9114f12fb473" (UID: "a5cb9eed-a657-4907-9765-9114f12fb473"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:05:10.124970 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:10.124944 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a5cb9eed-a657-4907-9765-9114f12fb473-istio-podinfo" pod "a5cb9eed-a657-4907-9765-9114f12fb473" (UID: "a5cb9eed-a657-4907-9765-9114f12fb473"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:05:10.204771 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:10.204727 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ae98fd5-ff16-44e3-80aa-139438c86c78-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ae98fd5-ff16-44e3-80aa-139438c86c78-istio-data\") pod \"1ae98fd5-ff16-44e3-80aa-139438c86c78\" (UID: \"1ae98fd5-ff16-44e3-80aa-139438c86c78\") " Oct 30 19:05:10.204954 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:10.204780 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ae98fd5-ff16-44e3-80aa-139438c86c78-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ae98fd5-ff16-44e3-80aa-139438c86c78-istio-envoy\") pod \"1ae98fd5-ff16-44e3-80aa-139438c86c78\" (UID: \"1ae98fd5-ff16-44e3-80aa-139438c86c78\") " Oct 30 19:05:10.204954 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:10.204853 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ae98fd5-ff16-44e3-80aa-139438c86c78-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ae98fd5-ff16-44e3-80aa-139438c86c78-workload-socket\") pod \"1ae98fd5-ff16-44e3-80aa-139438c86c78\" (UID: \"1ae98fd5-ff16-44e3-80aa-139438c86c78\") " Oct 30 19:05:10.204954 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:10.204947 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ae98fd5-ff16-44e3-80aa-139438c86c78-kube-api-access-87bxp\" (UniqueName: \"kubernetes.io/projected/1ae98fd5-ff16-44e3-80aa-139438c86c78-kube-api-access-87bxp\") pod \"1ae98fd5-ff16-44e3-80aa-139438c86c78\" (UID: \"1ae98fd5-ff16-44e3-80aa-139438c86c78\") " Oct 30 19:05:10.205310 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:10.205053 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ae98fd5-ff16-44e3-80aa-139438c86c78-istio-data" pod "1ae98fd5-ff16-44e3-80aa-139438c86c78" (UID: "1ae98fd5-ff16-44e3-80aa-139438c86c78"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:05:10.205310 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:10.205119 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ae98fd5-ff16-44e3-80aa-139438c86c78-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ae98fd5-ff16-44e3-80aa-139438c86c78-credential-socket\") pod \"1ae98fd5-ff16-44e3-80aa-139438c86c78\" (UID: \"1ae98fd5-ff16-44e3-80aa-139438c86c78\") " Oct 30 19:05:10.205310 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:10.205212 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1ae98fd5-ff16-44e3-80aa-139438c86c78-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ae98fd5-ff16-44e3-80aa-139438c86c78-istio-podinfo\") pod \"1ae98fd5-ff16-44e3-80aa-139438c86c78\" (UID: \"1ae98fd5-ff16-44e3-80aa-139438c86c78\") " Oct 30 19:05:10.205310 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:10.205245 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ae98fd5-ff16-44e3-80aa-139438c86c78-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ae98fd5-ff16-44e3-80aa-139438c86c78-workload-certs\") pod \"1ae98fd5-ff16-44e3-80aa-139438c86c78\" (UID: \"1ae98fd5-ff16-44e3-80aa-139438c86c78\") " Oct 30 19:05:10.205310 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:10.205282 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ae98fd5-ff16-44e3-80aa-139438c86c78-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ae98fd5-ff16-44e3-80aa-139438c86c78-istiod-ca-cert\") pod \"1ae98fd5-ff16-44e3-80aa-139438c86c78\" (UID: \"1ae98fd5-ff16-44e3-80aa-139438c86c78\") " Oct 30 19:05:10.205821 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:10.205394 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ae98fd5-ff16-44e3-80aa-139438c86c78-istio-token\" (UniqueName: \"kubernetes.io/projected/1ae98fd5-ff16-44e3-80aa-139438c86c78-istio-token\") pod \"1ae98fd5-ff16-44e3-80aa-139438c86c78\" (UID: \"1ae98fd5-ff16-44e3-80aa-139438c86c78\") " Oct 30 19:05:10.205821 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:10.205242 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ae98fd5-ff16-44e3-80aa-139438c86c78-workload-socket" pod "1ae98fd5-ff16-44e3-80aa-139438c86c78" (UID: "1ae98fd5-ff16-44e3-80aa-139438c86c78"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:05:10.205821 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:10.205480 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ae98fd5-ff16-44e3-80aa-139438c86c78-credential-socket" pod "1ae98fd5-ff16-44e3-80aa-139438c86c78" (UID: "1ae98fd5-ff16-44e3-80aa-139438c86c78"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:05:10.205821 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:10.205683 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ae98fd5-ff16-44e3-80aa-139438c86c78-workload-certs" pod "1ae98fd5-ff16-44e3-80aa-139438c86c78" (UID: "1ae98fd5-ff16-44e3-80aa-139438c86c78"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:05:10.206067 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:10.206044 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ae98fd5-ff16-44e3-80aa-139438c86c78-istiod-ca-cert" pod "1ae98fd5-ff16-44e3-80aa-139438c86c78" (UID: "1ae98fd5-ff16-44e3-80aa-139438c86c78"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:05:10.206658 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:10.206455 6960 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5cb9eed-a657-4907-9765-9114f12fb473-istio-podinfo\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:05:10.206658 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:10.206482 6960 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5cb9eed-a657-4907-9765-9114f12fb473-workload-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:05:10.206658 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:10.206498 6960 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5cb9eed-a657-4907-9765-9114f12fb473-istio-data\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:05:10.206658 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:10.206512 6960 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5cb9eed-a657-4907-9765-9114f12fb473-credential-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:05:10.206658 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:10.206526 6960 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ae98fd5-ff16-44e3-80aa-139438c86c78-istio-data\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:05:10.206658 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:10.206540 6960 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ae98fd5-ff16-44e3-80aa-139438c86c78-workload-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:05:10.206658 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:10.206553 6960 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5cb9eed-a657-4907-9765-9114f12fb473-istio-envoy\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:05:10.206658 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:10.206567 6960 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5cb9eed-a657-4907-9765-9114f12fb473-workload-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:05:10.206658 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:10.206580 6960 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5cb9eed-a657-4907-9765-9114f12fb473-istiod-ca-cert\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:05:10.206658 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:10.206594 6960 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ae98fd5-ff16-44e3-80aa-139438c86c78-credential-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:05:10.206658 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:10.206608 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5p2bm\" (UniqueName: \"kubernetes.io/projected/a5cb9eed-a657-4907-9765-9114f12fb473-kube-api-access-5p2bm\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:05:10.207498 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:10.206622 6960 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5cb9eed-a657-4907-9765-9114f12fb473-istio-token\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:05:10.207498 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:10.207460 6960 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ae98fd5-ff16-44e3-80aa-139438c86c78-workload-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:05:10.207498 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:10.207476 6960 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ae98fd5-ff16-44e3-80aa-139438c86c78-istiod-ca-cert\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:05:10.208629 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:10.208604 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ae98fd5-ff16-44e3-80aa-139438c86c78-istio-envoy" pod "1ae98fd5-ff16-44e3-80aa-139438c86c78" (UID: "1ae98fd5-ff16-44e3-80aa-139438c86c78"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:05:10.209013 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:10.208931 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ae98fd5-ff16-44e3-80aa-139438c86c78-istio-token" pod "1ae98fd5-ff16-44e3-80aa-139438c86c78" (UID: "1ae98fd5-ff16-44e3-80aa-139438c86c78"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:05:10.210025 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:10.209996 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ae98fd5-ff16-44e3-80aa-139438c86c78-kube-api-access-87bxp" pod "1ae98fd5-ff16-44e3-80aa-139438c86c78" (UID: "1ae98fd5-ff16-44e3-80aa-139438c86c78"). InnerVolumeSpecName "kube-api-access-87bxp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:05:10.210103 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:10.210023 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1ae98fd5-ff16-44e3-80aa-139438c86c78-istio-podinfo" pod "1ae98fd5-ff16-44e3-80aa-139438c86c78" (UID: "1ae98fd5-ff16-44e3-80aa-139438c86c78"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:05:10.308699 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:10.308665 6960 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ae98fd5-ff16-44e3-80aa-139438c86c78-istio-envoy\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:05:10.308699 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:10.308696 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87bxp\" (UniqueName: \"kubernetes.io/projected/1ae98fd5-ff16-44e3-80aa-139438c86c78-kube-api-access-87bxp\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:05:10.308699 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:10.308706 6960 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ae98fd5-ff16-44e3-80aa-139438c86c78-istio-podinfo\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:05:10.309124 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:10.308715 6960 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ae98fd5-ff16-44e3-80aa-139438c86c78-istio-token\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:05:10.386463 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:10.386386 6960 generic.go:356] "Generic (PLEG): container finished" podID="1ae98fd5-ff16-44e3-80aa-139438c86c78" containerID="31b0e3829260bd99c17c01905cb36463411dd72de7203f1e0db0ded1f8915fdc" exitCode=0 Oct 30 19:05:10.386632 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:10.386497 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-drlj6" Oct 30 19:05:10.386632 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:10.386529 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-drlj6" event={"ID":"1ae98fd5-ff16-44e3-80aa-139438c86c78","Type":"ContainerDied","Data":"31b0e3829260bd99c17c01905cb36463411dd72de7203f1e0db0ded1f8915fdc"} Oct 30 19:05:10.386632 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:10.386587 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-drlj6" event={"ID":"1ae98fd5-ff16-44e3-80aa-139438c86c78","Type":"ContainerDied","Data":"747b2286bc9d222fa9f90370687483162138ca9657021df88a184f80a9f5e881"} Oct 30 19:05:10.386632 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:10.386617 6960 scope.go:122] "RemoveContainer" containerID="31b0e3829260bd99c17c01905cb36463411dd72de7203f1e0db0ded1f8915fdc" Oct 30 19:05:10.388675 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:10.388617 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e8c554-6dcfccfdb5-wcdsr" event={"ID":"3e8ff0f2-eaf3-434d-b877-f95964be8923","Type":"ContainerStarted","Data":"70dd633a4adfbcfc39f6f44ad2d153fddee66f59cb5f70475811b6f714aa15e0"} Oct 30 19:05:10.390503 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:10.390480 6960 generic.go:356] "Generic (PLEG): container finished" podID="a5cb9eed-a657-4907-9765-9114f12fb473" containerID="4b5fc94da92519effb48045968ef28abe79fa131cd16f3f914cac50f186e7cc5" exitCode=0 Oct 30 19:05:10.390621 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:10.390550 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-frrjm" event={"ID":"a5cb9eed-a657-4907-9765-9114f12fb473","Type":"ContainerDied","Data":"4b5fc94da92519effb48045968ef28abe79fa131cd16f3f914cac50f186e7cc5"} Oct 30 19:05:10.390621 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:10.390577 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-frrjm" Oct 30 19:05:10.390621 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:10.390590 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-frrjm" event={"ID":"a5cb9eed-a657-4907-9765-9114f12fb473","Type":"ContainerDied","Data":"5c6ab46f016829d5812e3a061b3433633607b298d86360c904dad6254cffeff7"} Oct 30 19:05:10.402254 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:10.402138 6960 scope.go:122] "RemoveContainer" containerID="31b0e3829260bd99c17c01905cb36463411dd72de7203f1e0db0ded1f8915fdc" Oct 30 19:05:10.402600 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:05:10.402574 6960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31b0e3829260bd99c17c01905cb36463411dd72de7203f1e0db0ded1f8915fdc\": container with ID starting with 31b0e3829260bd99c17c01905cb36463411dd72de7203f1e0db0ded1f8915fdc not found: ID does not exist" containerID="31b0e3829260bd99c17c01905cb36463411dd72de7203f1e0db0ded1f8915fdc" Oct 30 19:05:10.402730 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:10.402706 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31b0e3829260bd99c17c01905cb36463411dd72de7203f1e0db0ded1f8915fdc"} err="failed to get container status \"31b0e3829260bd99c17c01905cb36463411dd72de7203f1e0db0ded1f8915fdc\": rpc error: code = NotFound desc = could not find container \"31b0e3829260bd99c17c01905cb36463411dd72de7203f1e0db0ded1f8915fdc\": container with ID starting with 31b0e3829260bd99c17c01905cb36463411dd72de7203f1e0db0ded1f8915fdc not found: ID does not exist" Oct 30 19:05:10.402819 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:10.402734 6960 scope.go:122] "RemoveContainer" containerID="4b5fc94da92519effb48045968ef28abe79fa131cd16f3f914cac50f186e7cc5" Oct 30 19:05:10.419775 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:10.418455 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-drlj6"] Oct 30 19:05:10.419775 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:10.419472 6960 scope.go:122] "RemoveContainer" containerID="4b5fc94da92519effb48045968ef28abe79fa131cd16f3f914cac50f186e7cc5" Oct 30 19:05:10.420272 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:05:10.420244 6960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b5fc94da92519effb48045968ef28abe79fa131cd16f3f914cac50f186e7cc5\": container with ID starting with 4b5fc94da92519effb48045968ef28abe79fa131cd16f3f914cac50f186e7cc5 not found: ID does not exist" containerID="4b5fc94da92519effb48045968ef28abe79fa131cd16f3f914cac50f186e7cc5" Oct 30 19:05:10.420370 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:10.420281 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b5fc94da92519effb48045968ef28abe79fa131cd16f3f914cac50f186e7cc5"} err="failed to get container status \"4b5fc94da92519effb48045968ef28abe79fa131cd16f3f914cac50f186e7cc5\": rpc error: code = NotFound desc = could not find container \"4b5fc94da92519effb48045968ef28abe79fa131cd16f3f914cac50f186e7cc5\": container with ID starting with 4b5fc94da92519effb48045968ef28abe79fa131cd16f3f914cac50f186e7cc5 not found: ID does not exist" Oct 30 19:05:10.426649 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:10.426627 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-drlj6"] Oct 30 19:05:10.440728 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:10.440706 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-frrjm"] Oct 30 19:05:10.444113 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:10.444091 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-frrjm"] Oct 30 19:05:10.595092 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:10.595062 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ae98fd5-ff16-44e3-80aa-139438c86c78" path="/var/lib/kubelet/pods/1ae98fd5-ff16-44e3-80aa-139438c86c78/volumes" Oct 30 19:05:10.595825 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:10.595802 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5cb9eed-a657-4907-9765-9114f12fb473" path="/var/lib/kubelet/pods/a5cb9eed-a657-4907-9765-9114f12fb473/volumes" Oct 30 19:05:10.603749 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:10.603701 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:10.972578 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:10.972542 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:11.123666 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:11.123632 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:11.399400 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:11.399360 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e8c554-6dcfccfdb5-wcdsr" event={"ID":"3e8ff0f2-eaf3-434d-b877-f95964be8923","Type":"ContainerStarted","Data":"553e0dc757254c4c3b47f581bc0972626a9af81c65d46adc6091b7bcada337c6"} Oct 30 19:05:11.399926 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:11.399601 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e8c554-6dcfccfdb5-wcdsr" Oct 30 19:05:11.425911 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:11.425846 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e8c554-6dcfccfdb5-wcdsr" podStartSLOduration=1.933121044 podStartE2EDuration="2.425826801s" totalImagesPullingTime="492.705757ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:05:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:05:11.423994341 +0000 UTC m=+1878.953025481" watchObservedRunningTime="2026-10-30 19:05:11.425826801 +0000 UTC m=+1878.954857939" Oct 30 19:05:11.658074 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:11.657986 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:12.250099 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:12.250055 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:12.304428 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:12.304392 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-d72beb-665d6f48c5-gb2vl"] Oct 30 19:05:12.305539 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:12.305514 6960 cpu_manager.go:414] "containerMap: removing container" podUID="1ae98fd5-ff16-44e3-80aa-139438c86c78" containerName="istio-proxy" Oct 30 19:05:12.305690 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:12.305542 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ae98fd5-ff16-44e3-80aa-139438c86c78" containerName="istio-proxy" Oct 30 19:05:12.305690 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:12.305577 6960 cpu_manager.go:414] "containerMap: removing container" podUID="a5cb9eed-a657-4907-9765-9114f12fb473" containerName="istio-proxy" Oct 30 19:05:12.305690 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:12.305585 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5cb9eed-a657-4907-9765-9114f12fb473" containerName="istio-proxy" Oct 30 19:05:12.305869 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:12.305759 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5cb9eed-a657-4907-9765-9114f12fb473" containerName="istio-proxy" Oct 30 19:05:12.305869 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:12.305789 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ae98fd5-ff16-44e3-80aa-139438c86c78" containerName="istio-proxy" Oct 30 19:05:12.311402 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:12.311383 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d72beb-665d6f48c5-gb2vl" Oct 30 19:05:12.317095 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:12.317057 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-d72beb-dockercfg-fgzzj\"" Oct 30 19:05:12.323994 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:12.323956 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-d72beb-665d6f48c5-gb2vl"] Oct 30 19:05:12.372088 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:12.371981 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:12.397885 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:12.397858 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-d72beb\"" Oct 30 19:05:12.513346 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:12.513245 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgq7m\" (UniqueName: \"kubernetes.io/projected/1ff59ea2-88a1-4532-a214-6bafc573b44f-kube-api-access-dgq7m\") pod \"payload-pre-processing-e2e-extmod-b-d72beb-665d6f48c5-gb2vl\" (UID: \"1ff59ea2-88a1-4532-a214-6bafc573b44f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d72beb-665d6f48c5-gb2vl" Oct 30 19:05:12.514014 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:12.513988 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ff59ea2-88a1-4532-a214-6bafc573b44f-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-d72beb-665d6f48c5-gb2vl\" (UID: \"1ff59ea2-88a1-4532-a214-6bafc573b44f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d72beb-665d6f48c5-gb2vl" Oct 30 19:05:12.622426 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:12.622391 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgq7m\" (UniqueName: \"kubernetes.io/projected/1ff59ea2-88a1-4532-a214-6bafc573b44f-kube-api-access-dgq7m\") pod \"payload-pre-processing-e2e-extmod-b-d72beb-665d6f48c5-gb2vl\" (UID: \"1ff59ea2-88a1-4532-a214-6bafc573b44f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d72beb-665d6f48c5-gb2vl" Oct 30 19:05:12.622592 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:12.622446 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ff59ea2-88a1-4532-a214-6bafc573b44f-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-d72beb-665d6f48c5-gb2vl\" (UID: \"1ff59ea2-88a1-4532-a214-6bafc573b44f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d72beb-665d6f48c5-gb2vl" Oct 30 19:05:12.624093 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:12.624053 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ff59ea2-88a1-4532-a214-6bafc573b44f-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-d72beb-665d6f48c5-gb2vl\" (UID: \"1ff59ea2-88a1-4532-a214-6bafc573b44f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d72beb-665d6f48c5-gb2vl" Oct 30 19:05:12.635574 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:12.635548 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgq7m\" (UniqueName: \"kubernetes.io/projected/1ff59ea2-88a1-4532-a214-6bafc573b44f-kube-api-access-dgq7m\") pod \"payload-pre-processing-e2e-extmod-b-d72beb-665d6f48c5-gb2vl\" (UID: \"1ff59ea2-88a1-4532-a214-6bafc573b44f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d72beb-665d6f48c5-gb2vl" Oct 30 19:05:12.808793 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:12.808763 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:12.930939 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:12.930908 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d72beb-665d6f48c5-gb2vl" Oct 30 19:05:13.244693 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:13.244662 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-d72beb-665d6f48c5-gb2vl"] Oct 30 19:05:13.244905 ip-10-0-140-163 kubenswrapper[6960]: W1030 19:05:13.244882 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ff59ea2_88a1_4532_a214_6bafc573b44f.slice/crio-264e21323ab8e261a4044bac58679c0268494a5ba420c116403e4242beace1a3 WatchSource:0}: Error finding container 264e21323ab8e261a4044bac58679c0268494a5ba420c116403e4242beace1a3: Status 404 returned error can't find the container with id 264e21323ab8e261a4044bac58679c0268494a5ba420c116403e4242beace1a3 Oct 30 19:05:13.424206 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:13.424169 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d72beb-665d6f48c5-gb2vl" event={"ID":"1ff59ea2-88a1-4532-a214-6bafc573b44f","Type":"ContainerStarted","Data":"19c1dd549e8a86947ba2838f42518490a12ad96f41061fa6311ffc51f75ad64c"} Oct 30 19:05:13.424393 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:13.424227 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d72beb-665d6f48c5-gb2vl" event={"ID":"1ff59ea2-88a1-4532-a214-6bafc573b44f","Type":"ContainerStarted","Data":"264e21323ab8e261a4044bac58679c0268494a5ba420c116403e4242beace1a3"} Oct 30 19:05:13.424393 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:13.424355 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d72beb-665d6f48c5-gb2vl" Oct 30 19:05:13.443748 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:13.443699 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d72beb-665d6f48c5-gb2vl" podStartSLOduration=1.4436816559999999 podStartE2EDuration="1.443681656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:05:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:05:13.442241639 +0000 UTC m=+1880.971272776" watchObservedRunningTime="2026-10-30 19:05:13.443681656 +0000 UTC m=+1880.972712798" Oct 30 19:05:13.796181 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:13.796128 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:14.434183 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:14.434134 6960 generic.go:356] "Generic (PLEG): container finished" podID="42b9fddf-ca5b-4181-8819-c32996d6ff0e" containerID="4de5bb8b562e6da61eaddd7604fb2e7e810953eaee3bed6dacf67fb993737fb6" exitCode=0 Oct 30 19:05:14.434183 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:14.434177 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d58l4" event={"ID":"42b9fddf-ca5b-4181-8819-c32996d6ff0e","Type":"ContainerDied","Data":"4de5bb8b562e6da61eaddd7604fb2e7e810953eaee3bed6dacf67fb993737fb6"} Oct 30 19:05:14.486342 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:14.486311 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-h72dd"] Oct 30 19:05:14.486909 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:14.486878 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-h72dd" podUID="4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b" containerName="istio-proxy" containerID="cri-o://ddb3577403de659337d418cae169967e2fbdabb6918860b2f17f55185ba3ee90" gracePeriod=30 Oct 30 19:05:14.582132 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:14.582106 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d58l4" Oct 30 19:05:14.630061 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:14.630022 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:14.651133 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:14.651105 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42b9fddf-ca5b-4181-8819-c32996d6ff0e-kube-api-access-8mcwp\" (UniqueName: \"kubernetes.io/projected/42b9fddf-ca5b-4181-8819-c32996d6ff0e-kube-api-access-8mcwp\") pod \"42b9fddf-ca5b-4181-8819-c32996d6ff0e\" (UID: \"42b9fddf-ca5b-4181-8819-c32996d6ff0e\") " Oct 30 19:05:14.651328 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:14.651195 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42b9fddf-ca5b-4181-8819-c32996d6ff0e-istio-token\" (UniqueName: \"kubernetes.io/projected/42b9fddf-ca5b-4181-8819-c32996d6ff0e-istio-token\") pod \"42b9fddf-ca5b-4181-8819-c32996d6ff0e\" (UID: \"42b9fddf-ca5b-4181-8819-c32996d6ff0e\") " Oct 30 19:05:14.651328 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:14.651264 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/42b9fddf-ca5b-4181-8819-c32996d6ff0e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42b9fddf-ca5b-4181-8819-c32996d6ff0e-istio-podinfo\") pod \"42b9fddf-ca5b-4181-8819-c32996d6ff0e\" (UID: \"42b9fddf-ca5b-4181-8819-c32996d6ff0e\") " Oct 30 19:05:14.651328 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:14.651320 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42b9fddf-ca5b-4181-8819-c32996d6ff0e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42b9fddf-ca5b-4181-8819-c32996d6ff0e-credential-socket\") pod \"42b9fddf-ca5b-4181-8819-c32996d6ff0e\" (UID: \"42b9fddf-ca5b-4181-8819-c32996d6ff0e\") " Oct 30 19:05:14.651500 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:14.651341 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42b9fddf-ca5b-4181-8819-c32996d6ff0e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42b9fddf-ca5b-4181-8819-c32996d6ff0e-workload-socket\") pod \"42b9fddf-ca5b-4181-8819-c32996d6ff0e\" (UID: \"42b9fddf-ca5b-4181-8819-c32996d6ff0e\") " Oct 30 19:05:14.651500 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:14.651374 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42b9fddf-ca5b-4181-8819-c32996d6ff0e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42b9fddf-ca5b-4181-8819-c32996d6ff0e-istio-envoy\") pod \"42b9fddf-ca5b-4181-8819-c32996d6ff0e\" (UID: \"42b9fddf-ca5b-4181-8819-c32996d6ff0e\") " Oct 30 19:05:14.651500 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:14.651442 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42b9fddf-ca5b-4181-8819-c32996d6ff0e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42b9fddf-ca5b-4181-8819-c32996d6ff0e-istiod-ca-cert\") pod \"42b9fddf-ca5b-4181-8819-c32996d6ff0e\" (UID: \"42b9fddf-ca5b-4181-8819-c32996d6ff0e\") " Oct 30 19:05:14.651500 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:14.651497 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42b9fddf-ca5b-4181-8819-c32996d6ff0e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42b9fddf-ca5b-4181-8819-c32996d6ff0e-workload-certs\") pod \"42b9fddf-ca5b-4181-8819-c32996d6ff0e\" (UID: \"42b9fddf-ca5b-4181-8819-c32996d6ff0e\") " Oct 30 19:05:14.651689 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:14.651526 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42b9fddf-ca5b-4181-8819-c32996d6ff0e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42b9fddf-ca5b-4181-8819-c32996d6ff0e-istio-data\") pod \"42b9fddf-ca5b-4181-8819-c32996d6ff0e\" (UID: \"42b9fddf-ca5b-4181-8819-c32996d6ff0e\") " Oct 30 19:05:14.651765 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:14.651729 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42b9fddf-ca5b-4181-8819-c32996d6ff0e-credential-socket" pod "42b9fddf-ca5b-4181-8819-c32996d6ff0e" (UID: "42b9fddf-ca5b-4181-8819-c32996d6ff0e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:05:14.652069 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:14.651993 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42b9fddf-ca5b-4181-8819-c32996d6ff0e-istio-data" pod "42b9fddf-ca5b-4181-8819-c32996d6ff0e" (UID: "42b9fddf-ca5b-4181-8819-c32996d6ff0e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:05:14.652531 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:14.652509 6960 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42b9fddf-ca5b-4181-8819-c32996d6ff0e-credential-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:05:14.652629 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:14.652538 6960 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42b9fddf-ca5b-4181-8819-c32996d6ff0e-istio-data\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:05:14.652629 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:14.652593 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42b9fddf-ca5b-4181-8819-c32996d6ff0e-istiod-ca-cert" pod "42b9fddf-ca5b-4181-8819-c32996d6ff0e" (UID: "42b9fddf-ca5b-4181-8819-c32996d6ff0e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:05:14.652745 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:14.652727 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42b9fddf-ca5b-4181-8819-c32996d6ff0e-workload-certs" pod "42b9fddf-ca5b-4181-8819-c32996d6ff0e" (UID: "42b9fddf-ca5b-4181-8819-c32996d6ff0e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:05:14.652948 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:14.652870 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42b9fddf-ca5b-4181-8819-c32996d6ff0e-workload-socket" pod "42b9fddf-ca5b-4181-8819-c32996d6ff0e" (UID: "42b9fddf-ca5b-4181-8819-c32996d6ff0e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:05:14.657539 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:14.656531 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42b9fddf-ca5b-4181-8819-c32996d6ff0e-kube-api-access-8mcwp" pod "42b9fddf-ca5b-4181-8819-c32996d6ff0e" (UID: "42b9fddf-ca5b-4181-8819-c32996d6ff0e"). InnerVolumeSpecName "kube-api-access-8mcwp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:05:14.657539 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:14.657487 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42b9fddf-ca5b-4181-8819-c32996d6ff0e-istio-envoy" pod "42b9fddf-ca5b-4181-8819-c32996d6ff0e" (UID: "42b9fddf-ca5b-4181-8819-c32996d6ff0e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:05:14.659098 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:14.659070 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42b9fddf-ca5b-4181-8819-c32996d6ff0e-istio-token" pod "42b9fddf-ca5b-4181-8819-c32996d6ff0e" (UID: "42b9fddf-ca5b-4181-8819-c32996d6ff0e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:05:14.660176 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:14.660135 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/42b9fddf-ca5b-4181-8819-c32996d6ff0e-istio-podinfo" pod "42b9fddf-ca5b-4181-8819-c32996d6ff0e" (UID: "42b9fddf-ca5b-4181-8819-c32996d6ff0e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:05:14.753844 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:14.753807 6960 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42b9fddf-ca5b-4181-8819-c32996d6ff0e-istiod-ca-cert\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:05:14.753844 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:14.753841 6960 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42b9fddf-ca5b-4181-8819-c32996d6ff0e-workload-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:05:14.753844 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:14.753854 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mcwp\" (UniqueName: \"kubernetes.io/projected/42b9fddf-ca5b-4181-8819-c32996d6ff0e-kube-api-access-8mcwp\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:05:14.754092 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:14.753864 6960 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42b9fddf-ca5b-4181-8819-c32996d6ff0e-istio-token\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:05:14.754092 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:14.753873 6960 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42b9fddf-ca5b-4181-8819-c32996d6ff0e-istio-podinfo\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:05:14.754092 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:14.753882 6960 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42b9fddf-ca5b-4181-8819-c32996d6ff0e-workload-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:05:14.754092 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:14.753890 6960 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42b9fddf-ca5b-4181-8819-c32996d6ff0e-istio-envoy\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:05:15.403069 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:15.403035 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:15.450711 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:15.450675 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d58l4" event={"ID":"42b9fddf-ca5b-4181-8819-c32996d6ff0e","Type":"ContainerDied","Data":"c85ab110636348dd86f24e56d79b459eb42ffc5b10de7aafc639e50fb4efe7ca"} Oct 30 19:05:15.450928 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:15.450747 6960 scope.go:122] "RemoveContainer" containerID="4de5bb8b562e6da61eaddd7604fb2e7e810953eaee3bed6dacf67fb993737fb6" Oct 30 19:05:15.450928 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:15.450922 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d58l4" Oct 30 19:05:15.486211 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:15.486176 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d58l4"] Oct 30 19:05:15.491343 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:15.491313 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-d58l4"] Oct 30 19:05:16.263730 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:16.263698 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-c9e9de-7d8cd96456-wzsgn"] Oct 30 19:05:16.264422 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:16.264399 6960 cpu_manager.go:414] "containerMap: removing container" podUID="42b9fddf-ca5b-4181-8819-c32996d6ff0e" containerName="istio-proxy" Oct 30 19:05:16.264422 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:16.264417 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42b9fddf-ca5b-4181-8819-c32996d6ff0e" containerName="istio-proxy" Oct 30 19:05:16.264630 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:16.264555 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="42b9fddf-ca5b-4181-8819-c32996d6ff0e" containerName="istio-proxy" Oct 30 19:05:16.269203 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:16.269173 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c9e9de-7d8cd96456-wzsgn" Oct 30 19:05:16.277644 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:16.277572 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-c9e9de-serving-cert\"" Oct 30 19:05:16.283196 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:16.280977 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-c9e9de-7d8cd96456-wzsgn"] Oct 30 19:05:16.356336 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:16.356305 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:16.384532 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:16.384493 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c861d5e-cd00-4c97-acda-9024e16bda48-openshift-service-ca\") pod \"maas-api-e2e-autores-c9e9de-7d8cd96456-wzsgn\" (UID: \"6c861d5e-cd00-4c97-acda-9024e16bda48\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c9e9de-7d8cd96456-wzsgn" Oct 30 19:05:16.384810 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:16.384675 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c861d5e-cd00-4c97-acda-9024e16bda48-metrics-certs\") pod \"maas-api-e2e-autores-c9e9de-7d8cd96456-wzsgn\" (UID: \"6c861d5e-cd00-4c97-acda-9024e16bda48\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c9e9de-7d8cd96456-wzsgn" Oct 30 19:05:16.384810 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:16.384751 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-md7lt\" (UniqueName: \"kubernetes.io/projected/6c861d5e-cd00-4c97-acda-9024e16bda48-kube-api-access-md7lt\") pod \"maas-api-e2e-autores-c9e9de-7d8cd96456-wzsgn\" (UID: \"6c861d5e-cd00-4c97-acda-9024e16bda48\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c9e9de-7d8cd96456-wzsgn" Oct 30 19:05:16.384927 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:16.384890 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c861d5e-cd00-4c97-acda-9024e16bda48-maas-api-tls\") pod \"maas-api-e2e-autores-c9e9de-7d8cd96456-wzsgn\" (UID: \"6c861d5e-cd00-4c97-acda-9024e16bda48\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c9e9de-7d8cd96456-wzsgn" Oct 30 19:05:16.472918 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:16.472877 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:16.486670 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:16.486626 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c861d5e-cd00-4c97-acda-9024e16bda48-maas-api-tls\") pod \"maas-api-e2e-autores-c9e9de-7d8cd96456-wzsgn\" (UID: \"6c861d5e-cd00-4c97-acda-9024e16bda48\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c9e9de-7d8cd96456-wzsgn" Oct 30 19:05:16.486891 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:16.486734 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c861d5e-cd00-4c97-acda-9024e16bda48-openshift-service-ca\") pod \"maas-api-e2e-autores-c9e9de-7d8cd96456-wzsgn\" (UID: \"6c861d5e-cd00-4c97-acda-9024e16bda48\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c9e9de-7d8cd96456-wzsgn" Oct 30 19:05:16.486969 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:16.486896 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c861d5e-cd00-4c97-acda-9024e16bda48-metrics-certs\") pod \"maas-api-e2e-autores-c9e9de-7d8cd96456-wzsgn\" (UID: \"6c861d5e-cd00-4c97-acda-9024e16bda48\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c9e9de-7d8cd96456-wzsgn" Oct 30 19:05:16.486969 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:16.486956 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-md7lt\" (UniqueName: \"kubernetes.io/projected/6c861d5e-cd00-4c97-acda-9024e16bda48-kube-api-access-md7lt\") pod \"maas-api-e2e-autores-c9e9de-7d8cd96456-wzsgn\" (UID: \"6c861d5e-cd00-4c97-acda-9024e16bda48\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c9e9de-7d8cd96456-wzsgn" Oct 30 19:05:16.487527 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:16.487503 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c861d5e-cd00-4c97-acda-9024e16bda48-openshift-service-ca\") pod \"maas-api-e2e-autores-c9e9de-7d8cd96456-wzsgn\" (UID: \"6c861d5e-cd00-4c97-acda-9024e16bda48\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c9e9de-7d8cd96456-wzsgn" Oct 30 19:05:16.489674 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:16.489649 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c861d5e-cd00-4c97-acda-9024e16bda48-maas-api-tls\") pod \"maas-api-e2e-autores-c9e9de-7d8cd96456-wzsgn\" (UID: \"6c861d5e-cd00-4c97-acda-9024e16bda48\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c9e9de-7d8cd96456-wzsgn" Oct 30 19:05:16.489800 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:16.489783 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c861d5e-cd00-4c97-acda-9024e16bda48-metrics-certs\") pod \"maas-api-e2e-autores-c9e9de-7d8cd96456-wzsgn\" (UID: \"6c861d5e-cd00-4c97-acda-9024e16bda48\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c9e9de-7d8cd96456-wzsgn" Oct 30 19:05:16.501715 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:16.501688 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-md7lt\" (UniqueName: \"kubernetes.io/projected/6c861d5e-cd00-4c97-acda-9024e16bda48-kube-api-access-md7lt\") pod \"maas-api-e2e-autores-c9e9de-7d8cd96456-wzsgn\" (UID: \"6c861d5e-cd00-4c97-acda-9024e16bda48\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c9e9de-7d8cd96456-wzsgn" Oct 30 19:05:16.591834 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:16.591801 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:16.592050 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:16.591865 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c9e9de-7d8cd96456-wzsgn" Oct 30 19:05:16.595256 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:16.595015 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42b9fddf-ca5b-4181-8819-c32996d6ff0e" path="/var/lib/kubelet/pods/42b9fddf-ca5b-4181-8819-c32996d6ff0e/volumes" Oct 30 19:05:16.930125 ip-10-0-140-163 kubenswrapper[6960]: W1030 19:05:16.918615 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c861d5e_cd00_4c97_acda_9024e16bda48.slice/crio-c21abcf7234d595bf025a69f75ff2e822a846effba12098829b061e8cd9ffcf8 WatchSource:0}: Error finding container c21abcf7234d595bf025a69f75ff2e822a846effba12098829b061e8cd9ffcf8: Status 404 returned error can't find the container with id c21abcf7234d595bf025a69f75ff2e822a846effba12098829b061e8cd9ffcf8 Oct 30 19:05:16.930125 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:16.919439 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-c9e9de-7d8cd96456-wzsgn"] Oct 30 19:05:16.985245 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:16.985204 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-5m6bs"] Oct 30 19:05:16.985688 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:16.985660 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-5m6bs" podUID="f0a610f7-6daf-48de-865d-a0788c9d036a" containerName="istio-proxy" containerID="cri-o://b93ffb29a8381e5b78b67b8c947c389bbfcf45da2e9590695e0c27f022c09885" gracePeriod=30 Oct 30 19:05:17.443307 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:17.443278 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e8c554-6dcfccfdb5-wcdsr" Oct 30 19:05:17.549978 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:17.549706 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c9e9de-7d8cd96456-wzsgn" event={"ID":"6c861d5e-cd00-4c97-acda-9024e16bda48","Type":"ContainerStarted","Data":"c21abcf7234d595bf025a69f75ff2e822a846effba12098829b061e8cd9ffcf8"} Oct 30 19:05:17.619536 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:17.617041 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:17.786552 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:17.786469 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:17.950452 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:17.950394 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:18.245494 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:18.245452 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:18.458527 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:18.458490 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:18.559331 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:18.559295 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c9e9de-7d8cd96456-wzsgn" event={"ID":"6c861d5e-cd00-4c97-acda-9024e16bda48","Type":"ContainerStarted","Data":"e1a84b0975259af23efe36a6bb026487112896994ec66f24d00e13661b9b633d"} Oct 30 19:05:18.559803 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:18.559363 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c9e9de-7d8cd96456-wzsgn" Oct 30 19:05:18.583523 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:18.583441 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c9e9de-7d8cd96456-wzsgn" podStartSLOduration=1.953840135 podStartE2EDuration="2.583423587s" totalImagesPullingTime="629.583452ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:05:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:05:18.582075066 +0000 UTC m=+1886.111106205" watchObservedRunningTime="2026-10-30 19:05:18.583423587 +0000 UTC m=+1886.112454725" Oct 30 19:05:18.643905 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:18.643864 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:18.792190 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:18.785836 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:18.993655 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:18.993568 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:19.312367 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:19.312321 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:19.460280 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:19.459998 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:19.872606 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:19.872539 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-h72dd" Oct 30 19:05:19.924244 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:19.924215 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b-kube-api-access-vxbnk\" (UniqueName: \"kubernetes.io/projected/4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b-kube-api-access-vxbnk\") pod \"4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b\" (UID: \"4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b\") " Oct 30 19:05:19.924440 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:19.924297 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b-credential-socket\") pod \"4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b\" (UID: \"4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b\") " Oct 30 19:05:19.924440 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:19.924361 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b-istiod-ca-cert\") pod \"4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b\" (UID: \"4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b\") " Oct 30 19:05:19.924440 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:19.924411 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b-workload-certs\") pod \"4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b\" (UID: \"4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b\") " Oct 30 19:05:19.924626 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:19.924494 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b-istio-token\" (UniqueName: \"kubernetes.io/projected/4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b-istio-token\") pod \"4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b\" (UID: \"4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b\") " Oct 30 19:05:19.924626 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:19.924531 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b-istio-podinfo\") pod \"4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b\" (UID: \"4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b\") " Oct 30 19:05:19.924738 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:19.924636 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b-istio-envoy\") pod \"4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b\" (UID: \"4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b\") " Oct 30 19:05:19.924738 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:19.924716 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b-workload-socket\") pod \"4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b\" (UID: \"4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b\") " Oct 30 19:05:19.924848 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:19.924776 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b-istio-data\") pod \"4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b\" (UID: \"4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b\") " Oct 30 19:05:19.924944 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:19.924629 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b-credential-socket" pod "4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b" (UID: "4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:05:19.925007 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:19.924915 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b-istiod-ca-cert" pod "4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b" (UID: "4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:05:19.925588 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:19.925254 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b-workload-socket" pod "4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b" (UID: "4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:05:19.925588 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:19.925504 6960 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b-workload-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:05:19.925588 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:19.925524 6960 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b-credential-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:05:19.925588 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:19.925539 6960 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b-istiod-ca-cert\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:05:19.925588 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:19.925568 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b-istio-data" pod "4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b" (UID: "4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:05:19.926006 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:19.925985 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b-workload-certs" pod "4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b" (UID: "4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:05:19.927678 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:19.927638 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b-istio-token" pod "4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b" (UID: "4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:05:19.939938 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:19.939913 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b-istio-podinfo" pod "4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b" (UID: "4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:05:19.941697 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:19.941661 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b-istio-envoy" pod "4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b" (UID: "4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:05:19.941775 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:19.941736 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b-kube-api-access-vxbnk" pod "4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b" (UID: "4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b"). InnerVolumeSpecName "kube-api-access-vxbnk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:05:20.027601 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:20.027571 6960 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b-istio-envoy\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:05:20.027601 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:20.027599 6960 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b-istio-data\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:05:20.027806 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:20.027609 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxbnk\" (UniqueName: \"kubernetes.io/projected/4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b-kube-api-access-vxbnk\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:05:20.027806 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:20.027621 6960 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b-workload-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:05:20.027806 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:20.027630 6960 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b-istio-token\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:05:20.027806 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:20.027638 6960 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b-istio-podinfo\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:05:20.177677 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:20.177604 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:20.304070 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:20.304007 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:20.586456 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:20.586422 6960 generic.go:356] "Generic (PLEG): container finished" podID="4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b" containerID="ddb3577403de659337d418cae169967e2fbdabb6918860b2f17f55185ba3ee90" exitCode=0 Oct 30 19:05:20.586658 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:20.586542 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-h72dd" Oct 30 19:05:20.596670 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:20.596295 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-h72dd" event={"ID":"4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b","Type":"ContainerDied","Data":"ddb3577403de659337d418cae169967e2fbdabb6918860b2f17f55185ba3ee90"} Oct 30 19:05:20.596670 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:20.596366 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-h72dd" event={"ID":"4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b","Type":"ContainerDied","Data":"5a3c8824468116a1a123f556e34b4a5ac822c45e113ef345d8445183db8a2fb3"} Oct 30 19:05:20.596670 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:20.596400 6960 scope.go:122] "RemoveContainer" containerID="ddb3577403de659337d418cae169967e2fbdabb6918860b2f17f55185ba3ee90" Oct 30 19:05:20.609025 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:20.609001 6960 scope.go:122] "RemoveContainer" containerID="ddb3577403de659337d418cae169967e2fbdabb6918860b2f17f55185ba3ee90" Oct 30 19:05:20.609383 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:05:20.609357 6960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddb3577403de659337d418cae169967e2fbdabb6918860b2f17f55185ba3ee90\": container with ID starting with ddb3577403de659337d418cae169967e2fbdabb6918860b2f17f55185ba3ee90 not found: ID does not exist" containerID="ddb3577403de659337d418cae169967e2fbdabb6918860b2f17f55185ba3ee90" Oct 30 19:05:20.609497 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:20.609389 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddb3577403de659337d418cae169967e2fbdabb6918860b2f17f55185ba3ee90"} err="failed to get container status \"ddb3577403de659337d418cae169967e2fbdabb6918860b2f17f55185ba3ee90\": rpc error: code = NotFound desc = could not find container \"ddb3577403de659337d418cae169967e2fbdabb6918860b2f17f55185ba3ee90\": container with ID starting with ddb3577403de659337d418cae169967e2fbdabb6918860b2f17f55185ba3ee90 not found: ID does not exist" Oct 30 19:05:20.617572 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:20.617540 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-h72dd"] Oct 30 19:05:20.624296 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:20.624270 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-h72dd"] Oct 30 19:05:21.135643 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:21.135602 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:22.114045 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:22.114008 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:22.299472 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:22.299446 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-5m6bs" Oct 30 19:05:22.364749 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:22.364719 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f0a610f7-6daf-48de-865d-a0788c9d036a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0a610f7-6daf-48de-865d-a0788c9d036a-istio-podinfo\") pod \"f0a610f7-6daf-48de-865d-a0788c9d036a\" (UID: \"f0a610f7-6daf-48de-865d-a0788c9d036a\") " Oct 30 19:05:22.364934 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:22.364859 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0a610f7-6daf-48de-865d-a0788c9d036a-istio-token\" (UniqueName: \"kubernetes.io/projected/f0a610f7-6daf-48de-865d-a0788c9d036a-istio-token\") pod \"f0a610f7-6daf-48de-865d-a0788c9d036a\" (UID: \"f0a610f7-6daf-48de-865d-a0788c9d036a\") " Oct 30 19:05:22.364934 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:22.364897 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0a610f7-6daf-48de-865d-a0788c9d036a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0a610f7-6daf-48de-865d-a0788c9d036a-workload-socket\") pod \"f0a610f7-6daf-48de-865d-a0788c9d036a\" (UID: \"f0a610f7-6daf-48de-865d-a0788c9d036a\") " Oct 30 19:05:22.365054 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:22.364997 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0a610f7-6daf-48de-865d-a0788c9d036a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0a610f7-6daf-48de-865d-a0788c9d036a-workload-certs\") pod \"f0a610f7-6daf-48de-865d-a0788c9d036a\" (UID: \"f0a610f7-6daf-48de-865d-a0788c9d036a\") " Oct 30 19:05:22.365109 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:22.365059 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0a610f7-6daf-48de-865d-a0788c9d036a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0a610f7-6daf-48de-865d-a0788c9d036a-istio-data\") pod \"f0a610f7-6daf-48de-865d-a0788c9d036a\" (UID: \"f0a610f7-6daf-48de-865d-a0788c9d036a\") " Oct 30 19:05:22.365109 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:22.365098 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0a610f7-6daf-48de-865d-a0788c9d036a-kube-api-access-rl2zr\" (UniqueName: \"kubernetes.io/projected/f0a610f7-6daf-48de-865d-a0788c9d036a-kube-api-access-rl2zr\") pod \"f0a610f7-6daf-48de-865d-a0788c9d036a\" (UID: \"f0a610f7-6daf-48de-865d-a0788c9d036a\") " Oct 30 19:05:22.365293 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:22.365189 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0a610f7-6daf-48de-865d-a0788c9d036a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0a610f7-6daf-48de-865d-a0788c9d036a-credential-socket\") pod \"f0a610f7-6daf-48de-865d-a0788c9d036a\" (UID: \"f0a610f7-6daf-48de-865d-a0788c9d036a\") " Oct 30 19:05:22.365293 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:22.365263 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0a610f7-6daf-48de-865d-a0788c9d036a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0a610f7-6daf-48de-865d-a0788c9d036a-istio-envoy\") pod \"f0a610f7-6daf-48de-865d-a0788c9d036a\" (UID: \"f0a610f7-6daf-48de-865d-a0788c9d036a\") " Oct 30 19:05:22.365418 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:22.365289 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0a610f7-6daf-48de-865d-a0788c9d036a-workload-socket" pod "f0a610f7-6daf-48de-865d-a0788c9d036a" (UID: "f0a610f7-6daf-48de-865d-a0788c9d036a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:05:22.365418 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:22.365375 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0a610f7-6daf-48de-865d-a0788c9d036a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0a610f7-6daf-48de-865d-a0788c9d036a-istiod-ca-cert\") pod \"f0a610f7-6daf-48de-865d-a0788c9d036a\" (UID: \"f0a610f7-6daf-48de-865d-a0788c9d036a\") " Oct 30 19:05:22.365627 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:22.365602 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0a610f7-6daf-48de-865d-a0788c9d036a-istio-data" pod "f0a610f7-6daf-48de-865d-a0788c9d036a" (UID: "f0a610f7-6daf-48de-865d-a0788c9d036a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:05:22.366127 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:22.366104 6960 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0a610f7-6daf-48de-865d-a0788c9d036a-istio-data\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:05:22.366297 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:22.366272 6960 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0a610f7-6daf-48de-865d-a0788c9d036a-workload-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:05:22.366297 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:22.366167 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0a610f7-6daf-48de-865d-a0788c9d036a-credential-socket" pod "f0a610f7-6daf-48de-865d-a0788c9d036a" (UID: "f0a610f7-6daf-48de-865d-a0788c9d036a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:05:22.366454 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:22.366192 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0a610f7-6daf-48de-865d-a0788c9d036a-istiod-ca-cert" pod "f0a610f7-6daf-48de-865d-a0788c9d036a" (UID: "f0a610f7-6daf-48de-865d-a0788c9d036a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:05:22.366454 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:22.366235 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0a610f7-6daf-48de-865d-a0788c9d036a-workload-certs" pod "f0a610f7-6daf-48de-865d-a0788c9d036a" (UID: "f0a610f7-6daf-48de-865d-a0788c9d036a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:05:22.367973 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:22.367947 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f0a610f7-6daf-48de-865d-a0788c9d036a-istio-podinfo" pod "f0a610f7-6daf-48de-865d-a0788c9d036a" (UID: "f0a610f7-6daf-48de-865d-a0788c9d036a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:05:22.368079 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:22.368029 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0a610f7-6daf-48de-865d-a0788c9d036a-istio-envoy" pod "f0a610f7-6daf-48de-865d-a0788c9d036a" (UID: "f0a610f7-6daf-48de-865d-a0788c9d036a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:05:22.368748 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:22.368721 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0a610f7-6daf-48de-865d-a0788c9d036a-kube-api-access-rl2zr" pod "f0a610f7-6daf-48de-865d-a0788c9d036a" (UID: "f0a610f7-6daf-48de-865d-a0788c9d036a"). InnerVolumeSpecName "kube-api-access-rl2zr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:05:22.368902 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:22.368882 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0a610f7-6daf-48de-865d-a0788c9d036a-istio-token" pod "f0a610f7-6daf-48de-865d-a0788c9d036a" (UID: "f0a610f7-6daf-48de-865d-a0788c9d036a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:05:22.467652 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:22.467618 6960 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0a610f7-6daf-48de-865d-a0788c9d036a-istio-envoy\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:05:22.467652 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:22.467645 6960 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0a610f7-6daf-48de-865d-a0788c9d036a-istiod-ca-cert\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:05:22.467652 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:22.467655 6960 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0a610f7-6daf-48de-865d-a0788c9d036a-istio-podinfo\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:05:22.467896 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:22.467665 6960 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0a610f7-6daf-48de-865d-a0788c9d036a-istio-token\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:05:22.467896 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:22.467675 6960 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0a610f7-6daf-48de-865d-a0788c9d036a-workload-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:05:22.467896 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:22.467684 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rl2zr\" (UniqueName: \"kubernetes.io/projected/f0a610f7-6daf-48de-865d-a0788c9d036a-kube-api-access-rl2zr\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:05:22.467896 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:22.467693 6960 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0a610f7-6daf-48de-865d-a0788c9d036a-credential-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:05:22.599702 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:22.595563 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b" path="/var/lib/kubelet/pods/4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b/volumes" Oct 30 19:05:22.613425 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:22.613392 6960 generic.go:356] "Generic (PLEG): container finished" podID="f0a610f7-6daf-48de-865d-a0788c9d036a" containerID="b93ffb29a8381e5b78b67b8c947c389bbfcf45da2e9590695e0c27f022c09885" exitCode=0 Oct 30 19:05:22.613532 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:22.613447 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-5m6bs" Oct 30 19:05:22.613532 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:22.613450 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-5m6bs" event={"ID":"f0a610f7-6daf-48de-865d-a0788c9d036a","Type":"ContainerDied","Data":"b93ffb29a8381e5b78b67b8c947c389bbfcf45da2e9590695e0c27f022c09885"} Oct 30 19:05:22.613532 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:22.613499 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-5m6bs" event={"ID":"f0a610f7-6daf-48de-865d-a0788c9d036a","Type":"ContainerDied","Data":"56dfee22b633efd8b22f4910a79a7b8a22ab7525881023d3889bb6ce3525160a"} Oct 30 19:05:22.613532 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:22.613519 6960 scope.go:122] "RemoveContainer" containerID="b93ffb29a8381e5b78b67b8c947c389bbfcf45da2e9590695e0c27f022c09885" Oct 30 19:05:22.625083 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:22.625067 6960 scope.go:122] "RemoveContainer" containerID="b93ffb29a8381e5b78b67b8c947c389bbfcf45da2e9590695e0c27f022c09885" Oct 30 19:05:22.625451 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:05:22.625432 6960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b93ffb29a8381e5b78b67b8c947c389bbfcf45da2e9590695e0c27f022c09885\": container with ID starting with b93ffb29a8381e5b78b67b8c947c389bbfcf45da2e9590695e0c27f022c09885 not found: ID does not exist" containerID="b93ffb29a8381e5b78b67b8c947c389bbfcf45da2e9590695e0c27f022c09885" Oct 30 19:05:22.625537 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:22.625460 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b93ffb29a8381e5b78b67b8c947c389bbfcf45da2e9590695e0c27f022c09885"} err="failed to get container status \"b93ffb29a8381e5b78b67b8c947c389bbfcf45da2e9590695e0c27f022c09885\": rpc error: code = NotFound desc = could not find container \"b93ffb29a8381e5b78b67b8c947c389bbfcf45da2e9590695e0c27f022c09885\": container with ID starting with b93ffb29a8381e5b78b67b8c947c389bbfcf45da2e9590695e0c27f022c09885 not found: ID does not exist" Oct 30 19:05:22.632644 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:22.632619 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-5m6bs"] Oct 30 19:05:22.637610 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:22.637586 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-5m6bs"] Oct 30 19:05:22.896056 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:22.895972 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:23.732342 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:23.732311 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:23.934405 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:05:23.934367 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 19:05:24.438107 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:24.438079 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d72beb-665d6f48c5-gb2vl" Oct 30 19:05:24.584119 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:24.584087 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c9e9de-7d8cd96456-wzsgn" Oct 30 19:05:24.593525 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:24.593497 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0a610f7-6daf-48de-865d-a0788c9d036a" path="/var/lib/kubelet/pods/f0a610f7-6daf-48de-865d-a0788c9d036a/volumes" Oct 30 19:05:24.642088 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:24.642052 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-de4bda-b48fd99d7-rw9vs"] Oct 30 19:05:24.643110 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:24.643084 6960 cpu_manager.go:414] "containerMap: removing container" podUID="f0a610f7-6daf-48de-865d-a0788c9d036a" containerName="istio-proxy" Oct 30 19:05:24.643110 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:24.643110 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0a610f7-6daf-48de-865d-a0788c9d036a" containerName="istio-proxy" Oct 30 19:05:24.643320 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:24.643203 6960 cpu_manager.go:414] "containerMap: removing container" podUID="4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b" containerName="istio-proxy" Oct 30 19:05:24.643320 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:24.643212 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b" containerName="istio-proxy" Oct 30 19:05:24.643425 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:24.643398 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cb85a2d-46a2-43e3-8d13-4cfc15f48c5b" containerName="istio-proxy" Oct 30 19:05:24.643425 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:24.643419 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0a610f7-6daf-48de-865d-a0788c9d036a" containerName="istio-proxy" Oct 30 19:05:24.648886 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:24.648867 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-de4bda-b48fd99d7-rw9vs" Oct 30 19:05:24.676986 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:24.676958 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-de4bda-b48fd99d7-rw9vs"] Oct 30 19:05:24.678931 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:24.678910 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-de4bda-serving-cert\"" Oct 30 19:05:24.698252 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:24.698172 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aef219d1-2021-4cea-b80b-ed995446c87c-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-de4bda-b48fd99d7-rw9vs\" (UID: \"aef219d1-2021-4cea-b80b-ed995446c87c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-de4bda-b48fd99d7-rw9vs" Oct 30 19:05:24.698371 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:24.698313 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aef219d1-2021-4cea-b80b-ed995446c87c-metrics-certs\") pod \"maas-api-e2e-extmod-a-de4bda-b48fd99d7-rw9vs\" (UID: \"aef219d1-2021-4cea-b80b-ed995446c87c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-de4bda-b48fd99d7-rw9vs" Oct 30 19:05:24.698698 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:24.698678 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwtzd\" (UniqueName: \"kubernetes.io/projected/aef219d1-2021-4cea-b80b-ed995446c87c-kube-api-access-kwtzd\") pod \"maas-api-e2e-extmod-a-de4bda-b48fd99d7-rw9vs\" (UID: \"aef219d1-2021-4cea-b80b-ed995446c87c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-de4bda-b48fd99d7-rw9vs" Oct 30 19:05:24.699012 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:24.698975 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aef219d1-2021-4cea-b80b-ed995446c87c-maas-api-tls\") pod \"maas-api-e2e-extmod-a-de4bda-b48fd99d7-rw9vs\" (UID: \"aef219d1-2021-4cea-b80b-ed995446c87c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-de4bda-b48fd99d7-rw9vs" Oct 30 19:05:24.732359 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:24.732324 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:24.799974 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:24.799943 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aef219d1-2021-4cea-b80b-ed995446c87c-metrics-certs\") pod \"maas-api-e2e-extmod-a-de4bda-b48fd99d7-rw9vs\" (UID: \"aef219d1-2021-4cea-b80b-ed995446c87c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-de4bda-b48fd99d7-rw9vs" Oct 30 19:05:24.800505 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:24.799983 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwtzd\" (UniqueName: \"kubernetes.io/projected/aef219d1-2021-4cea-b80b-ed995446c87c-kube-api-access-kwtzd\") pod \"maas-api-e2e-extmod-a-de4bda-b48fd99d7-rw9vs\" (UID: \"aef219d1-2021-4cea-b80b-ed995446c87c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-de4bda-b48fd99d7-rw9vs" Oct 30 19:05:24.800505 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:24.800099 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aef219d1-2021-4cea-b80b-ed995446c87c-maas-api-tls\") pod \"maas-api-e2e-extmod-a-de4bda-b48fd99d7-rw9vs\" (UID: \"aef219d1-2021-4cea-b80b-ed995446c87c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-de4bda-b48fd99d7-rw9vs" Oct 30 19:05:24.800505 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:24.800302 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aef219d1-2021-4cea-b80b-ed995446c87c-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-de4bda-b48fd99d7-rw9vs\" (UID: \"aef219d1-2021-4cea-b80b-ed995446c87c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-de4bda-b48fd99d7-rw9vs" Oct 30 19:05:24.801293 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:24.801268 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aef219d1-2021-4cea-b80b-ed995446c87c-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-de4bda-b48fd99d7-rw9vs\" (UID: \"aef219d1-2021-4cea-b80b-ed995446c87c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-de4bda-b48fd99d7-rw9vs" Oct 30 19:05:24.803196 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:24.803168 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aef219d1-2021-4cea-b80b-ed995446c87c-maas-api-tls\") pod \"maas-api-e2e-extmod-a-de4bda-b48fd99d7-rw9vs\" (UID: \"aef219d1-2021-4cea-b80b-ed995446c87c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-de4bda-b48fd99d7-rw9vs" Oct 30 19:05:24.803311 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:24.803259 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aef219d1-2021-4cea-b80b-ed995446c87c-metrics-certs\") pod \"maas-api-e2e-extmod-a-de4bda-b48fd99d7-rw9vs\" (UID: \"aef219d1-2021-4cea-b80b-ed995446c87c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-de4bda-b48fd99d7-rw9vs" Oct 30 19:05:24.820947 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:24.816521 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwtzd\" (UniqueName: \"kubernetes.io/projected/aef219d1-2021-4cea-b80b-ed995446c87c-kube-api-access-kwtzd\") pod \"maas-api-e2e-extmod-a-de4bda-b48fd99d7-rw9vs\" (UID: \"aef219d1-2021-4cea-b80b-ed995446c87c\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-de4bda-b48fd99d7-rw9vs" Oct 30 19:05:24.844847 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:24.844757 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:24.965854 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:24.965758 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-de4bda-b48fd99d7-rw9vs" Oct 30 19:05:25.260008 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:25.259926 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-de4bda-b48fd99d7-rw9vs"] Oct 30 19:05:25.265304 ip-10-0-140-163 kubenswrapper[6960]: W1030 19:05:25.265278 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaef219d1_2021_4cea_b80b_ed995446c87c.slice/crio-44fe3f4e80c797cf2cde371799bb96ddab1d23b28303569360894697a0c19913 WatchSource:0}: Error finding container 44fe3f4e80c797cf2cde371799bb96ddab1d23b28303569360894697a0c19913: Status 404 returned error can't find the container with id 44fe3f4e80c797cf2cde371799bb96ddab1d23b28303569360894697a0c19913 Oct 30 19:05:25.629115 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:25.629082 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:25.637919 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:25.637891 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-de4bda-b48fd99d7-rw9vs" event={"ID":"aef219d1-2021-4cea-b80b-ed995446c87c","Type":"ContainerStarted","Data":"44fe3f4e80c797cf2cde371799bb96ddab1d23b28303569360894697a0c19913"} Oct 30 19:05:26.421296 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:26.421266 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:26.647414 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:26.646280 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-de4bda-b48fd99d7-rw9vs" event={"ID":"aef219d1-2021-4cea-b80b-ed995446c87c","Type":"ContainerStarted","Data":"16338eae5e234525201a1f6d81fd58ccf458643ca6bee41dde9c5f375b6d74ce"} Oct 30 19:05:26.647414 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:26.646949 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-de4bda-b48fd99d7-rw9vs" Oct 30 19:05:26.674237 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:26.674111 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-de4bda-b48fd99d7-rw9vs" podStartSLOduration=2.204621301 podStartE2EDuration="2.674097991s" totalImagesPullingTime="469.47669ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:05:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:05:26.672486452 +0000 UTC m=+1894.201517591" watchObservedRunningTime="2026-10-30 19:05:26.674097991 +0000 UTC m=+1894.203129128" Oct 30 19:05:27.362036 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:27.361987 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:28.135761 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:28.135725 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:28.999949 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:28.999912 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:29.932444 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:29.932414 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:30.822152 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:30.822114 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:31.541841 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:31.541803 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-d72beb-97d7945b8-lf4k8"] Oct 30 19:05:31.547836 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:31.547808 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d72beb-97d7945b8-lf4k8" Oct 30 19:05:31.647134 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:31.647108 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-d72beb-serving-cert\"" Oct 30 19:05:31.651931 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:31.651893 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff4a8ee1-e818-4f1b-9b1e-f024237e3ec4-metrics-certs\") pod \"maas-api-e2e-extmod-b-d72beb-97d7945b8-lf4k8\" (UID: \"ff4a8ee1-e818-4f1b-9b1e-f024237e3ec4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d72beb-97d7945b8-lf4k8" Oct 30 19:05:31.652106 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:31.652055 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48nrp\" (UniqueName: \"kubernetes.io/projected/ff4a8ee1-e818-4f1b-9b1e-f024237e3ec4-kube-api-access-48nrp\") pod \"maas-api-e2e-extmod-b-d72beb-97d7945b8-lf4k8\" (UID: \"ff4a8ee1-e818-4f1b-9b1e-f024237e3ec4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d72beb-97d7945b8-lf4k8" Oct 30 19:05:31.652203 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:31.652112 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff4a8ee1-e818-4f1b-9b1e-f024237e3ec4-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-d72beb-97d7945b8-lf4k8\" (UID: \"ff4a8ee1-e818-4f1b-9b1e-f024237e3ec4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d72beb-97d7945b8-lf4k8" Oct 30 19:05:31.652278 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:31.652266 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff4a8ee1-e818-4f1b-9b1e-f024237e3ec4-maas-api-tls\") pod \"maas-api-e2e-extmod-b-d72beb-97d7945b8-lf4k8\" (UID: \"ff4a8ee1-e818-4f1b-9b1e-f024237e3ec4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d72beb-97d7945b8-lf4k8" Oct 30 19:05:31.662890 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:31.662861 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:31.674300 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:31.674279 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-d72beb-97d7945b8-lf4k8"] Oct 30 19:05:31.754550 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:31.754507 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff4a8ee1-e818-4f1b-9b1e-f024237e3ec4-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-d72beb-97d7945b8-lf4k8\" (UID: \"ff4a8ee1-e818-4f1b-9b1e-f024237e3ec4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d72beb-97d7945b8-lf4k8" Oct 30 19:05:31.754734 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:31.754591 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff4a8ee1-e818-4f1b-9b1e-f024237e3ec4-maas-api-tls\") pod \"maas-api-e2e-extmod-b-d72beb-97d7945b8-lf4k8\" (UID: \"ff4a8ee1-e818-4f1b-9b1e-f024237e3ec4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d72beb-97d7945b8-lf4k8" Oct 30 19:05:31.754786 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:05:31.754753 6960 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-d72beb-serving-cert: secret "maas-api-e2e-extmod-b-d72beb-serving-cert" not found Oct 30 19:05:31.754842 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:31.754820 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff4a8ee1-e818-4f1b-9b1e-f024237e3ec4-metrics-certs\") pod \"maas-api-e2e-extmod-b-d72beb-97d7945b8-lf4k8\" (UID: \"ff4a8ee1-e818-4f1b-9b1e-f024237e3ec4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d72beb-97d7945b8-lf4k8" Oct 30 19:05:31.754903 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:05:31.754843 6960 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff4a8ee1-e818-4f1b-9b1e-f024237e3ec4-maas-api-tls podName:ff4a8ee1-e818-4f1b-9b1e-f024237e3ec4 nodeName:}" failed. No retries permitted until 2026-10-30 19:05:32.254824315 +0000 UTC m=+1899.783855446 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ff4a8ee1-e818-4f1b-9b1e-f024237e3ec4-maas-api-tls") pod "maas-api-e2e-extmod-b-d72beb-97d7945b8-lf4k8" (UID: "ff4a8ee1-e818-4f1b-9b1e-f024237e3ec4") : secret "maas-api-e2e-extmod-b-d72beb-serving-cert" not found Oct 30 19:05:31.755101 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:31.755077 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48nrp\" (UniqueName: \"kubernetes.io/projected/ff4a8ee1-e818-4f1b-9b1e-f024237e3ec4-kube-api-access-48nrp\") pod \"maas-api-e2e-extmod-b-d72beb-97d7945b8-lf4k8\" (UID: \"ff4a8ee1-e818-4f1b-9b1e-f024237e3ec4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d72beb-97d7945b8-lf4k8" Oct 30 19:05:31.755514 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:31.755489 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff4a8ee1-e818-4f1b-9b1e-f024237e3ec4-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-d72beb-97d7945b8-lf4k8\" (UID: \"ff4a8ee1-e818-4f1b-9b1e-f024237e3ec4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d72beb-97d7945b8-lf4k8" Oct 30 19:05:31.757739 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:31.757720 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff4a8ee1-e818-4f1b-9b1e-f024237e3ec4-metrics-certs\") pod \"maas-api-e2e-extmod-b-d72beb-97d7945b8-lf4k8\" (UID: \"ff4a8ee1-e818-4f1b-9b1e-f024237e3ec4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d72beb-97d7945b8-lf4k8" Oct 30 19:05:31.769361 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:31.769330 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48nrp\" (UniqueName: \"kubernetes.io/projected/ff4a8ee1-e818-4f1b-9b1e-f024237e3ec4-kube-api-access-48nrp\") pod \"maas-api-e2e-extmod-b-d72beb-97d7945b8-lf4k8\" (UID: \"ff4a8ee1-e818-4f1b-9b1e-f024237e3ec4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d72beb-97d7945b8-lf4k8" Oct 30 19:05:31.774831 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:31.774783 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:32.162382 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:32.162345 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:32.262609 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:32.262574 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff4a8ee1-e818-4f1b-9b1e-f024237e3ec4-maas-api-tls\") pod \"maas-api-e2e-extmod-b-d72beb-97d7945b8-lf4k8\" (UID: \"ff4a8ee1-e818-4f1b-9b1e-f024237e3ec4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d72beb-97d7945b8-lf4k8" Oct 30 19:05:32.291506 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:32.290741 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff4a8ee1-e818-4f1b-9b1e-f024237e3ec4-maas-api-tls\") pod \"maas-api-e2e-extmod-b-d72beb-97d7945b8-lf4k8\" (UID: \"ff4a8ee1-e818-4f1b-9b1e-f024237e3ec4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d72beb-97d7945b8-lf4k8" Oct 30 19:05:32.303165 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:32.298390 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d72beb-97d7945b8-lf4k8" Oct 30 19:05:32.644911 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:32.644882 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-d72beb-97d7945b8-lf4k8"] Oct 30 19:05:32.645470 ip-10-0-140-163 kubenswrapper[6960]: W1030 19:05:32.645445 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff4a8ee1_e818_4f1b_9b1e_f024237e3ec4.slice/crio-eb4cb155966038d38f9529341952fe698f3036db56974b5a177cbb039c7a6605 WatchSource:0}: Error finding container eb4cb155966038d38f9529341952fe698f3036db56974b5a177cbb039c7a6605: Status 404 returned error can't find the container with id eb4cb155966038d38f9529341952fe698f3036db56974b5a177cbb039c7a6605 Oct 30 19:05:32.661939 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:32.661888 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-de4bda-b48fd99d7-rw9vs" Oct 30 19:05:32.707455 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:32.707375 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d72beb-97d7945b8-lf4k8" event={"ID":"ff4a8ee1-e818-4f1b-9b1e-f024237e3ec4","Type":"ContainerStarted","Data":"eb4cb155966038d38f9529341952fe698f3036db56974b5a177cbb039c7a6605"} Oct 30 19:05:32.755572 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:32.755524 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:33.717554 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:33.717520 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d72beb-97d7945b8-lf4k8" event={"ID":"ff4a8ee1-e818-4f1b-9b1e-f024237e3ec4","Type":"ContainerStarted","Data":"ff8db13b62e33c1494f560c71d3163b3765b81c54856ec89f0a27d0179b4a014"} Oct 30 19:05:33.718022 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:33.717576 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d72beb-97d7945b8-lf4k8" Oct 30 19:05:33.743599 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:33.743538 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d72beb-97d7945b8-lf4k8" podStartSLOduration=2.30419108 podStartE2EDuration="2.743521597s" totalImagesPullingTime="439.330517ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:05:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:05:33.74145158 +0000 UTC m=+1901.270482717" watchObservedRunningTime="2026-10-30 19:05:33.743521597 +0000 UTC m=+1901.272552735" Oct 30 19:05:34.056018 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:34.055977 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:34.307359 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:34.307261 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:35.011190 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:35.011128 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:35.659825 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:35.659789 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:35.924071 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:35.923987 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:36.155927 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:36.155891 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:36.557462 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:36.557426 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:36.931799 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:36.931696 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:37.742181 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:37.742108 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:38.594838 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:38.594788 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:39.597598 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:39.597568 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:39.736815 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:39.736786 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d72beb-97d7945b8-lf4k8" Oct 30 19:05:39.825975 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:39.825942 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:40.078037 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:40.078009 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:40.732026 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:40.731989 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:41.036506 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:41.036420 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:41.165185 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:41.165130 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:41.284410 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:41.284342 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:41.799480 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:41.799442 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:42.679834 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:42.679782 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:43.017772 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:43.017676 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:43.617822 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:43.617786 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:44.465308 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:44.465273 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:45.380064 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:45.380032 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:46.230349 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:46.230307 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:47.355179 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:47.355105 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:48.297413 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:48.297378 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:49.223178 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:49.223118 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:49.462255 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:49.462220 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:50.316553 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:50.316514 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:50.930806 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:50.930736 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:51.531926 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:51.531894 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:51.816280 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:51.816243 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:52.572624 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:52.572586 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:52.848418 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:52.848330 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:52.963897 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:52.963860 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:53.533119 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:53.533092 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:53.935699 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:05:53.935666 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 19:05:54.608541 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:54.608508 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:55.590136 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:55.590100 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:56.710176 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:56.710123 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:56.852587 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:56.852547 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:56.994213 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:56.993660 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:57.922662 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:57.922623 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:58.096200 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:58.087086 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:58.252870 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:58.252755 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:58.547128 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:58.547027 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:58.706311 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:58.706279 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:58.830805 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:58.830776 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:59.075768 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:59.075733 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:59.209629 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:59.209543 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:59.442264 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:59.442112 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:05:59.677171 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:05:59.676444 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:00.402936 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:00.402899 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:00.560725 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:00.560692 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:01.595935 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:01.595901 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:02.638565 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:02.638524 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:03.440947 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:03.440909 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-4e94aa-57fcc56d98-cpjh8"] Oct 30 19:06:03.448551 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:03.448526 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4e94aa-57fcc56d98-cpjh8" Oct 30 19:06:03.479457 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:03.479437 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-4e94aa-serving-cert\"" Oct 30 19:06:03.488336 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:03.488313 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-4e94aa-57fcc56d98-cpjh8"] Oct 30 19:06:03.507910 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:03.507882 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtx2r\" (UniqueName: \"kubernetes.io/projected/68ec602e-60f3-4540-bfcf-9b29cdcfda2f-kube-api-access-wtx2r\") pod \"maas-api-e2e-extmod-b-4e94aa-57fcc56d98-cpjh8\" (UID: \"68ec602e-60f3-4540-bfcf-9b29cdcfda2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4e94aa-57fcc56d98-cpjh8" Oct 30 19:06:03.508062 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:03.507951 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68ec602e-60f3-4540-bfcf-9b29cdcfda2f-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-4e94aa-57fcc56d98-cpjh8\" (UID: \"68ec602e-60f3-4540-bfcf-9b29cdcfda2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4e94aa-57fcc56d98-cpjh8" Oct 30 19:06:03.508235 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:03.508206 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68ec602e-60f3-4540-bfcf-9b29cdcfda2f-maas-api-tls\") pod \"maas-api-e2e-extmod-b-4e94aa-57fcc56d98-cpjh8\" (UID: \"68ec602e-60f3-4540-bfcf-9b29cdcfda2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4e94aa-57fcc56d98-cpjh8" Oct 30 19:06:03.508325 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:03.508255 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68ec602e-60f3-4540-bfcf-9b29cdcfda2f-metrics-certs\") pod \"maas-api-e2e-extmod-b-4e94aa-57fcc56d98-cpjh8\" (UID: \"68ec602e-60f3-4540-bfcf-9b29cdcfda2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4e94aa-57fcc56d98-cpjh8" Oct 30 19:06:03.609296 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:03.609261 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtx2r\" (UniqueName: \"kubernetes.io/projected/68ec602e-60f3-4540-bfcf-9b29cdcfda2f-kube-api-access-wtx2r\") pod \"maas-api-e2e-extmod-b-4e94aa-57fcc56d98-cpjh8\" (UID: \"68ec602e-60f3-4540-bfcf-9b29cdcfda2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4e94aa-57fcc56d98-cpjh8" Oct 30 19:06:03.609460 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:03.609345 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68ec602e-60f3-4540-bfcf-9b29cdcfda2f-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-4e94aa-57fcc56d98-cpjh8\" (UID: \"68ec602e-60f3-4540-bfcf-9b29cdcfda2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4e94aa-57fcc56d98-cpjh8" Oct 30 19:06:03.609460 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:03.609430 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68ec602e-60f3-4540-bfcf-9b29cdcfda2f-maas-api-tls\") pod \"maas-api-e2e-extmod-b-4e94aa-57fcc56d98-cpjh8\" (UID: \"68ec602e-60f3-4540-bfcf-9b29cdcfda2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4e94aa-57fcc56d98-cpjh8" Oct 30 19:06:03.609460 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:03.609451 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68ec602e-60f3-4540-bfcf-9b29cdcfda2f-metrics-certs\") pod \"maas-api-e2e-extmod-b-4e94aa-57fcc56d98-cpjh8\" (UID: \"68ec602e-60f3-4540-bfcf-9b29cdcfda2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4e94aa-57fcc56d98-cpjh8" Oct 30 19:06:03.609992 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:03.609968 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68ec602e-60f3-4540-bfcf-9b29cdcfda2f-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-4e94aa-57fcc56d98-cpjh8\" (UID: \"68ec602e-60f3-4540-bfcf-9b29cdcfda2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4e94aa-57fcc56d98-cpjh8" Oct 30 19:06:03.612344 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:03.612320 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68ec602e-60f3-4540-bfcf-9b29cdcfda2f-metrics-certs\") pod \"maas-api-e2e-extmod-b-4e94aa-57fcc56d98-cpjh8\" (UID: \"68ec602e-60f3-4540-bfcf-9b29cdcfda2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4e94aa-57fcc56d98-cpjh8" Oct 30 19:06:03.612437 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:03.612320 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68ec602e-60f3-4540-bfcf-9b29cdcfda2f-maas-api-tls\") pod \"maas-api-e2e-extmod-b-4e94aa-57fcc56d98-cpjh8\" (UID: \"68ec602e-60f3-4540-bfcf-9b29cdcfda2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4e94aa-57fcc56d98-cpjh8" Oct 30 19:06:03.627930 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:03.627898 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtx2r\" (UniqueName: \"kubernetes.io/projected/68ec602e-60f3-4540-bfcf-9b29cdcfda2f-kube-api-access-wtx2r\") pod \"maas-api-e2e-extmod-b-4e94aa-57fcc56d98-cpjh8\" (UID: \"68ec602e-60f3-4540-bfcf-9b29cdcfda2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4e94aa-57fcc56d98-cpjh8" Oct 30 19:06:03.642685 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:03.642644 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:03.752817 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:03.752733 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:03.763623 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:03.763599 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4e94aa-57fcc56d98-cpjh8" Oct 30 19:06:04.030598 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:04.030491 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:04.072071 ip-10-0-140-163 kubenswrapper[6960]: W1030 19:06:04.071943 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68ec602e_60f3_4540_bfcf_9b29cdcfda2f.slice/crio-32fefe5550c9d2a47f3ae966c487627dcc9602a8746086ed4d48e0f1713127f9 WatchSource:0}: Error finding container 32fefe5550c9d2a47f3ae966c487627dcc9602a8746086ed4d48e0f1713127f9: Status 404 returned error can't find the container with id 32fefe5550c9d2a47f3ae966c487627dcc9602a8746086ed4d48e0f1713127f9 Oct 30 19:06:04.072233 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:04.072208 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-4e94aa-57fcc56d98-cpjh8"] Oct 30 19:06:04.810349 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:04.810312 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:04.992999 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:04.992912 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4e94aa-57fcc56d98-cpjh8" event={"ID":"68ec602e-60f3-4540-bfcf-9b29cdcfda2f","Type":"ContainerStarted","Data":"0ff79a08593d1f82155641332c3043c38f9e26a23ac94031466d0441a5de5282"} Oct 30 19:06:04.992999 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:04.992974 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4e94aa-57fcc56d98-cpjh8" event={"ID":"68ec602e-60f3-4540-bfcf-9b29cdcfda2f","Type":"ContainerStarted","Data":"32fefe5550c9d2a47f3ae966c487627dcc9602a8746086ed4d48e0f1713127f9"} Oct 30 19:06:04.992999 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:04.992998 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4e94aa-57fcc56d98-cpjh8" Oct 30 19:06:05.017649 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:05.017606 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4e94aa-57fcc56d98-cpjh8" podStartSLOduration=1.497703317 podStartE2EDuration="2.017591984s" totalImagesPullingTime="519.888667ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:06:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:06:05.013785517 +0000 UTC m=+1932.542816654" watchObservedRunningTime="2026-10-30 19:06:05.017591984 +0000 UTC m=+1932.546623144" Oct 30 19:06:05.817894 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:05.817825 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:06.800366 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:06.800317 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:07.662454 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:07.662417 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:08.540125 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:08.540095 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:09.410489 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:09.410444 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:10.299277 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:10.299235 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:11.005359 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:11.005332 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4e94aa-57fcc56d98-cpjh8" Oct 30 19:06:11.383365 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:11.383325 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:11.927734 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:11.927697 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:12.811431 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:12.811389 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:13.607324 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:13.607287 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:14.426428 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:14.426396 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:14.547724 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:14.547695 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:15.217533 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:15.217488 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-x95gb"] Oct 30 19:06:15.222368 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:15.222340 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-x95gb" Oct 30 19:06:15.244401 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:15.244340 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-x95gb"] Oct 30 19:06:15.317772 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:15.317747 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-de4bda-openshift-default-dockercfg-jchfn\"" Oct 30 19:06:15.331155 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:15.331110 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:15.406322 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:15.406287 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94b983c1-08fc-4f62-85de-e24696834c7c-istiod-ca-cert\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-x95gb\" (UID: \"94b983c1-08fc-4f62-85de-e24696834c7c\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-x95gb" Oct 30 19:06:15.406543 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:15.406434 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94b983c1-08fc-4f62-85de-e24696834c7c-istio-data\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-x95gb\" (UID: \"94b983c1-08fc-4f62-85de-e24696834c7c\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-x95gb" Oct 30 19:06:15.406543 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:15.406502 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94b983c1-08fc-4f62-85de-e24696834c7c-workload-certs\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-x95gb\" (UID: \"94b983c1-08fc-4f62-85de-e24696834c7c\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-x95gb" Oct 30 19:06:15.406723 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:15.406697 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cqq4\" (UniqueName: \"kubernetes.io/projected/94b983c1-08fc-4f62-85de-e24696834c7c-kube-api-access-5cqq4\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-x95gb\" (UID: \"94b983c1-08fc-4f62-85de-e24696834c7c\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-x95gb" Oct 30 19:06:15.406763 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:15.406752 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94b983c1-08fc-4f62-85de-e24696834c7c-credential-socket\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-x95gb\" (UID: \"94b983c1-08fc-4f62-85de-e24696834c7c\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-x95gb" Oct 30 19:06:15.406811 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:15.406796 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94b983c1-08fc-4f62-85de-e24696834c7c-istio-envoy\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-x95gb\" (UID: \"94b983c1-08fc-4f62-85de-e24696834c7c\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-x95gb" Oct 30 19:06:15.406880 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:15.406865 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94b983c1-08fc-4f62-85de-e24696834c7c-workload-socket\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-x95gb\" (UID: \"94b983c1-08fc-4f62-85de-e24696834c7c\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-x95gb" Oct 30 19:06:15.406919 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:15.406910 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94b983c1-08fc-4f62-85de-e24696834c7c-istio-token\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-x95gb\" (UID: \"94b983c1-08fc-4f62-85de-e24696834c7c\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-x95gb" Oct 30 19:06:15.407008 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:15.406997 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94b983c1-08fc-4f62-85de-e24696834c7c-istio-podinfo\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-x95gb\" (UID: \"94b983c1-08fc-4f62-85de-e24696834c7c\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-x95gb" Oct 30 19:06:15.508751 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:15.508675 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94b983c1-08fc-4f62-85de-e24696834c7c-istio-data\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-x95gb\" (UID: \"94b983c1-08fc-4f62-85de-e24696834c7c\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-x95gb" Oct 30 19:06:15.508751 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:15.508728 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94b983c1-08fc-4f62-85de-e24696834c7c-workload-certs\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-x95gb\" (UID: \"94b983c1-08fc-4f62-85de-e24696834c7c\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-x95gb" Oct 30 19:06:15.509310 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:15.508804 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cqq4\" (UniqueName: \"kubernetes.io/projected/94b983c1-08fc-4f62-85de-e24696834c7c-kube-api-access-5cqq4\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-x95gb\" (UID: \"94b983c1-08fc-4f62-85de-e24696834c7c\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-x95gb" Oct 30 19:06:15.509310 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:15.508831 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94b983c1-08fc-4f62-85de-e24696834c7c-credential-socket\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-x95gb\" (UID: \"94b983c1-08fc-4f62-85de-e24696834c7c\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-x95gb" Oct 30 19:06:15.509310 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:15.508865 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94b983c1-08fc-4f62-85de-e24696834c7c-istio-envoy\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-x95gb\" (UID: \"94b983c1-08fc-4f62-85de-e24696834c7c\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-x95gb" Oct 30 19:06:15.509310 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:15.508904 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94b983c1-08fc-4f62-85de-e24696834c7c-workload-socket\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-x95gb\" (UID: \"94b983c1-08fc-4f62-85de-e24696834c7c\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-x95gb" Oct 30 19:06:15.509310 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:15.508999 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94b983c1-08fc-4f62-85de-e24696834c7c-istio-token\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-x95gb\" (UID: \"94b983c1-08fc-4f62-85de-e24696834c7c\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-x95gb" Oct 30 19:06:15.509310 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:15.509108 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94b983c1-08fc-4f62-85de-e24696834c7c-istio-data\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-x95gb\" (UID: \"94b983c1-08fc-4f62-85de-e24696834c7c\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-x95gb" Oct 30 19:06:15.509310 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:15.509126 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94b983c1-08fc-4f62-85de-e24696834c7c-istio-podinfo\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-x95gb\" (UID: \"94b983c1-08fc-4f62-85de-e24696834c7c\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-x95gb" Oct 30 19:06:15.509310 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:15.509179 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94b983c1-08fc-4f62-85de-e24696834c7c-workload-certs\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-x95gb\" (UID: \"94b983c1-08fc-4f62-85de-e24696834c7c\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-x95gb" Oct 30 19:06:15.509310 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:15.509192 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94b983c1-08fc-4f62-85de-e24696834c7c-istiod-ca-cert\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-x95gb\" (UID: \"94b983c1-08fc-4f62-85de-e24696834c7c\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-x95gb" Oct 30 19:06:15.509310 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:15.509241 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94b983c1-08fc-4f62-85de-e24696834c7c-workload-socket\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-x95gb\" (UID: \"94b983c1-08fc-4f62-85de-e24696834c7c\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-x95gb" Oct 30 19:06:15.509740 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:15.509614 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94b983c1-08fc-4f62-85de-e24696834c7c-credential-socket\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-x95gb\" (UID: \"94b983c1-08fc-4f62-85de-e24696834c7c\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-x95gb" Oct 30 19:06:15.509850 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:15.509806 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94b983c1-08fc-4f62-85de-e24696834c7c-istiod-ca-cert\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-x95gb\" (UID: \"94b983c1-08fc-4f62-85de-e24696834c7c\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-x95gb" Oct 30 19:06:15.511468 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:15.511444 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94b983c1-08fc-4f62-85de-e24696834c7c-istio-envoy\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-x95gb\" (UID: \"94b983c1-08fc-4f62-85de-e24696834c7c\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-x95gb" Oct 30 19:06:15.511732 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:15.511716 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94b983c1-08fc-4f62-85de-e24696834c7c-istio-podinfo\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-x95gb\" (UID: \"94b983c1-08fc-4f62-85de-e24696834c7c\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-x95gb" Oct 30 19:06:15.521743 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:15.521721 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94b983c1-08fc-4f62-85de-e24696834c7c-istio-token\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-x95gb\" (UID: \"94b983c1-08fc-4f62-85de-e24696834c7c\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-x95gb" Oct 30 19:06:15.521849 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:15.521824 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cqq4\" (UniqueName: \"kubernetes.io/projected/94b983c1-08fc-4f62-85de-e24696834c7c-kube-api-access-5cqq4\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-x95gb\" (UID: \"94b983c1-08fc-4f62-85de-e24696834c7c\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-x95gb" Oct 30 19:06:15.545471 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:15.545447 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-x95gb" Oct 30 19:06:15.833527 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:15.833474 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-x95gb"] Oct 30 19:06:15.834018 ip-10-0-140-163 kubenswrapper[6960]: W1030 19:06:15.833988 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94b983c1_08fc_4f62_85de_e24696834c7c.slice/crio-d03f8f09adcde2ed208f1fc13fe18aeb9e67a404147df5f815432504ccc5c115 WatchSource:0}: Error finding container d03f8f09adcde2ed208f1fc13fe18aeb9e67a404147df5f815432504ccc5c115: Status 404 returned error can't find the container with id d03f8f09adcde2ed208f1fc13fe18aeb9e67a404147df5f815432504ccc5c115 Oct 30 19:06:15.837351 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:15.837314 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 19:06:15.837447 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:15.837402 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 19:06:15.837489 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:15.837455 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 19:06:16.074785 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:16.074745 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-x95gb" event={"ID":"94b983c1-08fc-4f62-85de-e24696834c7c","Type":"ContainerStarted","Data":"2f742c3a4ba0c9b04c13be9851d0c9eaa29e63a527939ae7663cfd119a048c66"} Oct 30 19:06:16.074983 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:16.074800 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-x95gb" event={"ID":"94b983c1-08fc-4f62-85de-e24696834c7c","Type":"ContainerStarted","Data":"d03f8f09adcde2ed208f1fc13fe18aeb9e67a404147df5f815432504ccc5c115"} Oct 30 19:06:16.098648 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:16.098529 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-x95gb" podStartSLOduration=1.098513287 podStartE2EDuration="1.098513287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:06:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:06:16.096592497 +0000 UTC m=+1943.625623635" watchObservedRunningTime="2026-10-30 19:06:16.098513287 +0000 UTC m=+1943.627544450" Oct 30 19:06:16.325219 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:16.325182 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:16.546723 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:16.546645 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-x95gb" Oct 30 19:06:16.548267 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:16.548244 6960 patch_prober.go:29] interesting pod/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-x95gb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 19:06:16.548362 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:16.548294 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-x95gb" podUID="94b983c1-08fc-4f62-85de-e24696834c7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 19:06:17.250543 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:17.250502 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:17.545893 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:17.545818 6960 patch_prober.go:29] interesting pod/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-x95gb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 19:06:17.545893 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:17.545866 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-x95gb" podUID="94b983c1-08fc-4f62-85de-e24696834c7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 19:06:18.343350 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:18.343320 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:18.546076 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:18.546044 6960 patch_prober.go:29] interesting pod/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-x95gb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 19:06:18.546276 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:18.546097 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-x95gb" podUID="94b983c1-08fc-4f62-85de-e24696834c7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 19:06:19.240006 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:19.239976 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-7c6027-7b8d4f4fb9-mpf4x"] Oct 30 19:06:19.245440 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:19.244870 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7c6027-7b8d4f4fb9-mpf4x" Oct 30 19:06:19.253183 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:19.251353 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-7c6027-serving-cert\"" Oct 30 19:06:19.291758 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:19.291718 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2276\" (UniqueName: \"kubernetes.io/projected/cd525e12-1c0c-4ab5-91ac-9028a847cab6-kube-api-access-h2276\") pod \"maas-api-e2e-extmod-b-7c6027-7b8d4f4fb9-mpf4x\" (UID: \"cd525e12-1c0c-4ab5-91ac-9028a847cab6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7c6027-7b8d4f4fb9-mpf4x" Oct 30 19:06:19.292218 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:19.292184 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd525e12-1c0c-4ab5-91ac-9028a847cab6-maas-api-tls\") pod \"maas-api-e2e-extmod-b-7c6027-7b8d4f4fb9-mpf4x\" (UID: \"cd525e12-1c0c-4ab5-91ac-9028a847cab6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7c6027-7b8d4f4fb9-mpf4x" Oct 30 19:06:19.292344 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:19.292249 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd525e12-1c0c-4ab5-91ac-9028a847cab6-metrics-certs\") pod \"maas-api-e2e-extmod-b-7c6027-7b8d4f4fb9-mpf4x\" (UID: \"cd525e12-1c0c-4ab5-91ac-9028a847cab6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7c6027-7b8d4f4fb9-mpf4x" Oct 30 19:06:19.292416 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:19.292368 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd525e12-1c0c-4ab5-91ac-9028a847cab6-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-7c6027-7b8d4f4fb9-mpf4x\" (UID: \"cd525e12-1c0c-4ab5-91ac-9028a847cab6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7c6027-7b8d4f4fb9-mpf4x" Oct 30 19:06:19.293967 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:19.293944 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-7c6027-7b8d4f4fb9-mpf4x"] Oct 30 19:06:19.306399 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:19.306361 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:19.414194 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:19.397618 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2276\" (UniqueName: \"kubernetes.io/projected/cd525e12-1c0c-4ab5-91ac-9028a847cab6-kube-api-access-h2276\") pod \"maas-api-e2e-extmod-b-7c6027-7b8d4f4fb9-mpf4x\" (UID: \"cd525e12-1c0c-4ab5-91ac-9028a847cab6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7c6027-7b8d4f4fb9-mpf4x" Oct 30 19:06:19.415827 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:19.415794 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd525e12-1c0c-4ab5-91ac-9028a847cab6-maas-api-tls\") pod \"maas-api-e2e-extmod-b-7c6027-7b8d4f4fb9-mpf4x\" (UID: \"cd525e12-1c0c-4ab5-91ac-9028a847cab6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7c6027-7b8d4f4fb9-mpf4x" Oct 30 19:06:19.416061 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:19.416042 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd525e12-1c0c-4ab5-91ac-9028a847cab6-metrics-certs\") pod \"maas-api-e2e-extmod-b-7c6027-7b8d4f4fb9-mpf4x\" (UID: \"cd525e12-1c0c-4ab5-91ac-9028a847cab6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7c6027-7b8d4f4fb9-mpf4x" Oct 30 19:06:19.429420 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:19.427264 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd525e12-1c0c-4ab5-91ac-9028a847cab6-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-7c6027-7b8d4f4fb9-mpf4x\" (UID: \"cd525e12-1c0c-4ab5-91ac-9028a847cab6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7c6027-7b8d4f4fb9-mpf4x" Oct 30 19:06:19.429420 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:06:19.415968 6960 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-7c6027-serving-cert: secret "maas-api-e2e-extmod-b-7c6027-serving-cert" not found Oct 30 19:06:19.429420 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:06:19.427487 6960 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd525e12-1c0c-4ab5-91ac-9028a847cab6-maas-api-tls podName:cd525e12-1c0c-4ab5-91ac-9028a847cab6 nodeName:}" failed. No retries permitted until 2026-10-30 19:06:19.92745871 +0000 UTC m=+1947.456489840 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cd525e12-1c0c-4ab5-91ac-9028a847cab6-maas-api-tls") pod "maas-api-e2e-extmod-b-7c6027-7b8d4f4fb9-mpf4x" (UID: "cd525e12-1c0c-4ab5-91ac-9028a847cab6") : secret "maas-api-e2e-extmod-b-7c6027-serving-cert" not found Oct 30 19:06:19.429420 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:19.427605 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd525e12-1c0c-4ab5-91ac-9028a847cab6-metrics-certs\") pod \"maas-api-e2e-extmod-b-7c6027-7b8d4f4fb9-mpf4x\" (UID: \"cd525e12-1c0c-4ab5-91ac-9028a847cab6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7c6027-7b8d4f4fb9-mpf4x" Oct 30 19:06:19.429420 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:19.428034 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd525e12-1c0c-4ab5-91ac-9028a847cab6-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-7c6027-7b8d4f4fb9-mpf4x\" (UID: \"cd525e12-1c0c-4ab5-91ac-9028a847cab6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7c6027-7b8d4f4fb9-mpf4x" Oct 30 19:06:19.429916 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:19.429899 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2276\" (UniqueName: \"kubernetes.io/projected/cd525e12-1c0c-4ab5-91ac-9028a847cab6-kube-api-access-h2276\") pod \"maas-api-e2e-extmod-b-7c6027-7b8d4f4fb9-mpf4x\" (UID: \"cd525e12-1c0c-4ab5-91ac-9028a847cab6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7c6027-7b8d4f4fb9-mpf4x" Oct 30 19:06:19.431761 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:19.431528 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:19.546513 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:19.546438 6960 patch_prober.go:29] interesting pod/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-x95gb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 19:06:19.546513 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:19.546483 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-x95gb" podUID="94b983c1-08fc-4f62-85de-e24696834c7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 19:06:19.699850 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:19.699732 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:19.936176 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:19.934317 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd525e12-1c0c-4ab5-91ac-9028a847cab6-maas-api-tls\") pod \"maas-api-e2e-extmod-b-7c6027-7b8d4f4fb9-mpf4x\" (UID: \"cd525e12-1c0c-4ab5-91ac-9028a847cab6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7c6027-7b8d4f4fb9-mpf4x" Oct 30 19:06:19.939125 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:19.937976 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd525e12-1c0c-4ab5-91ac-9028a847cab6-maas-api-tls\") pod \"maas-api-e2e-extmod-b-7c6027-7b8d4f4fb9-mpf4x\" (UID: \"cd525e12-1c0c-4ab5-91ac-9028a847cab6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7c6027-7b8d4f4fb9-mpf4x" Oct 30 19:06:20.201260 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:20.201175 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7c6027-7b8d4f4fb9-mpf4x" Oct 30 19:06:20.314453 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:20.314420 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-cvflz"] Oct 30 19:06:20.326682 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:20.326452 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-cvflz"] Oct 30 19:06:20.326682 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:20.326583 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-cvflz" Oct 30 19:06:20.346443 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:20.346412 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb8a7572-d545-404a-bb45-19afdbe8274d-istio-envoy\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-cvflz\" (UID: \"eb8a7572-d545-404a-bb45-19afdbe8274d\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-cvflz" Oct 30 19:06:20.346577 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:20.346466 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb8a7572-d545-404a-bb45-19afdbe8274d-workload-socket\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-cvflz\" (UID: \"eb8a7572-d545-404a-bb45-19afdbe8274d\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-cvflz" Oct 30 19:06:20.346577 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:20.346520 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb8a7572-d545-404a-bb45-19afdbe8274d-istiod-ca-cert\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-cvflz\" (UID: \"eb8a7572-d545-404a-bb45-19afdbe8274d\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-cvflz" Oct 30 19:06:20.346716 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:20.346688 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb8a7572-d545-404a-bb45-19afdbe8274d-credential-socket\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-cvflz\" (UID: \"eb8a7572-d545-404a-bb45-19afdbe8274d\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-cvflz" Oct 30 19:06:20.346780 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:20.346766 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tm6k4\" (UniqueName: \"kubernetes.io/projected/eb8a7572-d545-404a-bb45-19afdbe8274d-kube-api-access-tm6k4\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-cvflz\" (UID: \"eb8a7572-d545-404a-bb45-19afdbe8274d\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-cvflz" Oct 30 19:06:20.346872 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:20.346843 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb8a7572-d545-404a-bb45-19afdbe8274d-istio-data\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-cvflz\" (UID: \"eb8a7572-d545-404a-bb45-19afdbe8274d\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-cvflz" Oct 30 19:06:20.346947 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:20.346914 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb8a7572-d545-404a-bb45-19afdbe8274d-istio-token\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-cvflz\" (UID: \"eb8a7572-d545-404a-bb45-19afdbe8274d\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-cvflz" Oct 30 19:06:20.347006 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:20.346966 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb8a7572-d545-404a-bb45-19afdbe8274d-workload-certs\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-cvflz\" (UID: \"eb8a7572-d545-404a-bb45-19afdbe8274d\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-cvflz" Oct 30 19:06:20.347006 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:20.346994 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb8a7572-d545-404a-bb45-19afdbe8274d-istio-podinfo\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-cvflz\" (UID: \"eb8a7572-d545-404a-bb45-19afdbe8274d\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-cvflz" Oct 30 19:06:20.387948 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:20.387113 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-c9e9de-openshift-default-dockercfg-ct2zv\"" Oct 30 19:06:20.447905 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:20.447877 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm6k4\" (UniqueName: \"kubernetes.io/projected/eb8a7572-d545-404a-bb45-19afdbe8274d-kube-api-access-tm6k4\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-cvflz\" (UID: \"eb8a7572-d545-404a-bb45-19afdbe8274d\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-cvflz" Oct 30 19:06:20.448413 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:20.447947 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb8a7572-d545-404a-bb45-19afdbe8274d-istio-data\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-cvflz\" (UID: \"eb8a7572-d545-404a-bb45-19afdbe8274d\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-cvflz" Oct 30 19:06:20.448413 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:20.448013 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb8a7572-d545-404a-bb45-19afdbe8274d-istio-token\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-cvflz\" (UID: \"eb8a7572-d545-404a-bb45-19afdbe8274d\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-cvflz" Oct 30 19:06:20.448413 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:20.448065 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb8a7572-d545-404a-bb45-19afdbe8274d-workload-certs\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-cvflz\" (UID: \"eb8a7572-d545-404a-bb45-19afdbe8274d\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-cvflz" Oct 30 19:06:20.448413 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:20.448094 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb8a7572-d545-404a-bb45-19afdbe8274d-istio-podinfo\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-cvflz\" (UID: \"eb8a7572-d545-404a-bb45-19afdbe8274d\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-cvflz" Oct 30 19:06:20.448413 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:20.448205 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb8a7572-d545-404a-bb45-19afdbe8274d-istio-envoy\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-cvflz\" (UID: \"eb8a7572-d545-404a-bb45-19afdbe8274d\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-cvflz" Oct 30 19:06:20.448413 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:20.448256 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb8a7572-d545-404a-bb45-19afdbe8274d-workload-socket\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-cvflz\" (UID: \"eb8a7572-d545-404a-bb45-19afdbe8274d\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-cvflz" Oct 30 19:06:20.448413 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:20.448313 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb8a7572-d545-404a-bb45-19afdbe8274d-istiod-ca-cert\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-cvflz\" (UID: \"eb8a7572-d545-404a-bb45-19afdbe8274d\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-cvflz" Oct 30 19:06:20.448413 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:20.448386 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb8a7572-d545-404a-bb45-19afdbe8274d-istio-data\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-cvflz\" (UID: \"eb8a7572-d545-404a-bb45-19afdbe8274d\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-cvflz" Oct 30 19:06:20.448720 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:20.448392 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb8a7572-d545-404a-bb45-19afdbe8274d-credential-socket\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-cvflz\" (UID: \"eb8a7572-d545-404a-bb45-19afdbe8274d\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-cvflz" Oct 30 19:06:20.449022 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:20.449006 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb8a7572-d545-404a-bb45-19afdbe8274d-credential-socket\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-cvflz\" (UID: \"eb8a7572-d545-404a-bb45-19afdbe8274d\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-cvflz" Oct 30 19:06:20.449134 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:20.449077 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb8a7572-d545-404a-bb45-19afdbe8274d-workload-certs\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-cvflz\" (UID: \"eb8a7572-d545-404a-bb45-19afdbe8274d\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-cvflz" Oct 30 19:06:20.449236 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:20.449075 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb8a7572-d545-404a-bb45-19afdbe8274d-istiod-ca-cert\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-cvflz\" (UID: \"eb8a7572-d545-404a-bb45-19afdbe8274d\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-cvflz" Oct 30 19:06:20.449299 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:20.449208 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb8a7572-d545-404a-bb45-19afdbe8274d-workload-socket\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-cvflz\" (UID: \"eb8a7572-d545-404a-bb45-19afdbe8274d\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-cvflz" Oct 30 19:06:20.450888 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:20.450869 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb8a7572-d545-404a-bb45-19afdbe8274d-istio-podinfo\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-cvflz\" (UID: \"eb8a7572-d545-404a-bb45-19afdbe8274d\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-cvflz" Oct 30 19:06:20.451239 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:20.451219 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb8a7572-d545-404a-bb45-19afdbe8274d-istio-envoy\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-cvflz\" (UID: \"eb8a7572-d545-404a-bb45-19afdbe8274d\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-cvflz" Oct 30 19:06:20.490988 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:20.490911 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tm6k4\" (UniqueName: \"kubernetes.io/projected/eb8a7572-d545-404a-bb45-19afdbe8274d-kube-api-access-tm6k4\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-cvflz\" (UID: \"eb8a7572-d545-404a-bb45-19afdbe8274d\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-cvflz" Oct 30 19:06:20.492380 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:20.492033 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb8a7572-d545-404a-bb45-19afdbe8274d-istio-token\") pod \"e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-cvflz\" (UID: \"eb8a7572-d545-404a-bb45-19afdbe8274d\") " pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-cvflz" Oct 30 19:06:20.510578 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:20.510556 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-7c6027-7b8d4f4fb9-mpf4x"] Oct 30 19:06:20.511993 ip-10-0-140-163 kubenswrapper[6960]: W1030 19:06:20.511969 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd525e12_1c0c_4ab5_91ac_9028a847cab6.slice/crio-8033161bcd4e953e4c574934874cd27fae4e4a0c0b04be4d0b6b6e866a51c0c5 WatchSource:0}: Error finding container 8033161bcd4e953e4c574934874cd27fae4e4a0c0b04be4d0b6b6e866a51c0c5: Status 404 returned error can't find the container with id 8033161bcd4e953e4c574934874cd27fae4e4a0c0b04be4d0b6b6e866a51c0c5 Oct 30 19:06:20.546811 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:20.546780 6960 patch_prober.go:29] interesting pod/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-x95gb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 19:06:20.546968 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:20.546831 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-x95gb" podUID="94b983c1-08fc-4f62-85de-e24696834c7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 19:06:20.631717 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:20.631687 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:20.671288 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:20.670403 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-cvflz" Oct 30 19:06:21.022510 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:21.022478 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-cvflz"] Oct 30 19:06:21.023366 ip-10-0-140-163 kubenswrapper[6960]: W1030 19:06:21.023345 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb8a7572_d545_404a_bb45_19afdbe8274d.slice/crio-bd58db3a469d6b474e373f718c06e3c3758ce6b722a084c4e1c1b1a50b2194f9 WatchSource:0}: Error finding container bd58db3a469d6b474e373f718c06e3c3758ce6b722a084c4e1c1b1a50b2194f9: Status 404 returned error can't find the container with id bd58db3a469d6b474e373f718c06e3c3758ce6b722a084c4e1c1b1a50b2194f9 Oct 30 19:06:21.026378 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:21.026350 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 19:06:21.026453 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:21.026420 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 19:06:21.026491 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:21.026476 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 19:06:21.196862 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:21.196740 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-cvflz" event={"ID":"eb8a7572-d545-404a-bb45-19afdbe8274d","Type":"ContainerStarted","Data":"bd58db3a469d6b474e373f718c06e3c3758ce6b722a084c4e1c1b1a50b2194f9"} Oct 30 19:06:21.199110 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:21.199090 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7c6027-7b8d4f4fb9-mpf4x" event={"ID":"cd525e12-1c0c-4ab5-91ac-9028a847cab6","Type":"ContainerStarted","Data":"8033161bcd4e953e4c574934874cd27fae4e4a0c0b04be4d0b6b6e866a51c0c5"} Oct 30 19:06:21.545909 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:21.545876 6960 patch_prober.go:29] interesting pod/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-x95gb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 19:06:21.546348 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:21.545934 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-x95gb" podUID="94b983c1-08fc-4f62-85de-e24696834c7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 19:06:21.633128 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:21.633082 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:21.819493 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:21.818543 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-xxv5m"] Oct 30 19:06:21.823878 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:21.823859 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-xxv5m" Oct 30 19:06:21.836609 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:21.836104 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-xxv5m"] Oct 30 19:06:21.902752 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:21.902720 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71138058-4eed-4e10-b831-50aa2d950f79-istiod-ca-cert\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-xxv5m\" (UID: \"71138058-4eed-4e10-b831-50aa2d950f79\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-xxv5m" Oct 30 19:06:21.902942 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:21.902902 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71138058-4eed-4e10-b831-50aa2d950f79-workload-socket\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-xxv5m\" (UID: \"71138058-4eed-4e10-b831-50aa2d950f79\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-xxv5m" Oct 30 19:06:21.903036 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:21.903013 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71138058-4eed-4e10-b831-50aa2d950f79-istio-token\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-xxv5m\" (UID: \"71138058-4eed-4e10-b831-50aa2d950f79\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-xxv5m" Oct 30 19:06:21.903133 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:21.903109 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71138058-4eed-4e10-b831-50aa2d950f79-istio-envoy\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-xxv5m\" (UID: \"71138058-4eed-4e10-b831-50aa2d950f79\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-xxv5m" Oct 30 19:06:21.903216 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:21.903177 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71138058-4eed-4e10-b831-50aa2d950f79-credential-socket\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-xxv5m\" (UID: \"71138058-4eed-4e10-b831-50aa2d950f79\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-xxv5m" Oct 30 19:06:21.903317 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:21.903298 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71138058-4eed-4e10-b831-50aa2d950f79-istio-data\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-xxv5m\" (UID: \"71138058-4eed-4e10-b831-50aa2d950f79\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-xxv5m" Oct 30 19:06:21.903374 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:21.903327 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71138058-4eed-4e10-b831-50aa2d950f79-workload-certs\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-xxv5m\" (UID: \"71138058-4eed-4e10-b831-50aa2d950f79\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-xxv5m" Oct 30 19:06:21.903433 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:21.903419 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bgg6\" (UniqueName: \"kubernetes.io/projected/71138058-4eed-4e10-b831-50aa2d950f79-kube-api-access-4bgg6\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-xxv5m\" (UID: \"71138058-4eed-4e10-b831-50aa2d950f79\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-xxv5m" Oct 30 19:06:21.903509 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:21.903494 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71138058-4eed-4e10-b831-50aa2d950f79-istio-podinfo\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-xxv5m\" (UID: \"71138058-4eed-4e10-b831-50aa2d950f79\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-xxv5m" Oct 30 19:06:21.919515 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:21.919485 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-d72beb-openshift-default-dockercfg-drvvh\"" Oct 30 19:06:22.004553 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:22.004508 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71138058-4eed-4e10-b831-50aa2d950f79-workload-socket\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-xxv5m\" (UID: \"71138058-4eed-4e10-b831-50aa2d950f79\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-xxv5m" Oct 30 19:06:22.004737 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:22.004584 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71138058-4eed-4e10-b831-50aa2d950f79-istio-token\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-xxv5m\" (UID: \"71138058-4eed-4e10-b831-50aa2d950f79\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-xxv5m" Oct 30 19:06:22.004737 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:22.004628 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71138058-4eed-4e10-b831-50aa2d950f79-istio-envoy\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-xxv5m\" (UID: \"71138058-4eed-4e10-b831-50aa2d950f79\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-xxv5m" Oct 30 19:06:22.004737 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:22.004667 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71138058-4eed-4e10-b831-50aa2d950f79-credential-socket\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-xxv5m\" (UID: \"71138058-4eed-4e10-b831-50aa2d950f79\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-xxv5m" Oct 30 19:06:22.004737 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:22.004711 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71138058-4eed-4e10-b831-50aa2d950f79-istio-data\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-xxv5m\" (UID: \"71138058-4eed-4e10-b831-50aa2d950f79\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-xxv5m" Oct 30 19:06:22.004737 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:22.004734 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71138058-4eed-4e10-b831-50aa2d950f79-workload-certs\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-xxv5m\" (UID: \"71138058-4eed-4e10-b831-50aa2d950f79\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-xxv5m" Oct 30 19:06:22.005033 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:22.004807 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bgg6\" (UniqueName: \"kubernetes.io/projected/71138058-4eed-4e10-b831-50aa2d950f79-kube-api-access-4bgg6\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-xxv5m\" (UID: \"71138058-4eed-4e10-b831-50aa2d950f79\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-xxv5m" Oct 30 19:06:22.005033 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:22.004884 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71138058-4eed-4e10-b831-50aa2d950f79-istio-podinfo\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-xxv5m\" (UID: \"71138058-4eed-4e10-b831-50aa2d950f79\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-xxv5m" Oct 30 19:06:22.005033 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:22.005019 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71138058-4eed-4e10-b831-50aa2d950f79-istiod-ca-cert\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-xxv5m\" (UID: \"71138058-4eed-4e10-b831-50aa2d950f79\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-xxv5m" Oct 30 19:06:22.005239 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:22.005043 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71138058-4eed-4e10-b831-50aa2d950f79-workload-socket\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-xxv5m\" (UID: \"71138058-4eed-4e10-b831-50aa2d950f79\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-xxv5m" Oct 30 19:06:22.005239 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:22.005098 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71138058-4eed-4e10-b831-50aa2d950f79-credential-socket\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-xxv5m\" (UID: \"71138058-4eed-4e10-b831-50aa2d950f79\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-xxv5m" Oct 30 19:06:22.005239 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:22.005228 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71138058-4eed-4e10-b831-50aa2d950f79-workload-certs\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-xxv5m\" (UID: \"71138058-4eed-4e10-b831-50aa2d950f79\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-xxv5m" Oct 30 19:06:22.005528 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:22.005503 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71138058-4eed-4e10-b831-50aa2d950f79-istio-data\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-xxv5m\" (UID: \"71138058-4eed-4e10-b831-50aa2d950f79\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-xxv5m" Oct 30 19:06:22.005775 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:22.005754 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71138058-4eed-4e10-b831-50aa2d950f79-istiod-ca-cert\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-xxv5m\" (UID: \"71138058-4eed-4e10-b831-50aa2d950f79\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-xxv5m" Oct 30 19:06:22.007886 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:22.007863 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71138058-4eed-4e10-b831-50aa2d950f79-istio-envoy\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-xxv5m\" (UID: \"71138058-4eed-4e10-b831-50aa2d950f79\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-xxv5m" Oct 30 19:06:22.008353 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:22.008333 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71138058-4eed-4e10-b831-50aa2d950f79-istio-podinfo\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-xxv5m\" (UID: \"71138058-4eed-4e10-b831-50aa2d950f79\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-xxv5m" Oct 30 19:06:22.017890 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:22.017844 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bgg6\" (UniqueName: \"kubernetes.io/projected/71138058-4eed-4e10-b831-50aa2d950f79-kube-api-access-4bgg6\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-xxv5m\" (UID: \"71138058-4eed-4e10-b831-50aa2d950f79\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-xxv5m" Oct 30 19:06:22.018469 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:22.018442 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71138058-4eed-4e10-b831-50aa2d950f79-istio-token\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-xxv5m\" (UID: \"71138058-4eed-4e10-b831-50aa2d950f79\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-xxv5m" Oct 30 19:06:22.149493 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:22.149397 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-xxv5m" Oct 30 19:06:22.218193 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:22.213884 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-cvflz" event={"ID":"eb8a7572-d545-404a-bb45-19afdbe8274d","Type":"ContainerStarted","Data":"c4e5966656ea077156dbdc2d0d060388d0eab8118b77c9e41accf0f16bb7a39a"} Oct 30 19:06:22.218193 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:22.216336 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7c6027-7b8d4f4fb9-mpf4x" event={"ID":"cd525e12-1c0c-4ab5-91ac-9028a847cab6","Type":"ContainerStarted","Data":"8941d022aae24244d7adb90695623dcb4e2427f5ecf04a4363c40ead06aaaf8e"} Oct 30 19:06:22.218193 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:22.216478 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7c6027-7b8d4f4fb9-mpf4x" Oct 30 19:06:22.238452 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:22.238398 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-cvflz" podStartSLOduration=2.238382143 podStartE2EDuration="2.238382143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:06:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:06:22.236417874 +0000 UTC m=+1949.765449013" watchObservedRunningTime="2026-10-30 19:06:22.238382143 +0000 UTC m=+1949.767413280" Oct 30 19:06:22.280425 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:22.277328 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7c6027-7b8d4f4fb9-mpf4x" podStartSLOduration=2.703472874 podStartE2EDuration="3.277305033s" totalImagesPullingTime="573.832159ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:06:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:06:22.267681605 +0000 UTC m=+1949.796712744" watchObservedRunningTime="2026-10-30 19:06:22.277305033 +0000 UTC m=+1949.806336173" Oct 30 19:06:22.489535 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:22.489420 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-xxv5m"] Oct 30 19:06:22.506299 ip-10-0-140-163 kubenswrapper[6960]: W1030 19:06:22.506250 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71138058_4eed_4e10_b831_50aa2d950f79.slice/crio-8a43e94145eef632458063af97b9a2e4b071e3ba9c19abd6df91bb46467d256e WatchSource:0}: Error finding container 8a43e94145eef632458063af97b9a2e4b071e3ba9c19abd6df91bb46467d256e: Status 404 returned error can't find the container with id 8a43e94145eef632458063af97b9a2e4b071e3ba9c19abd6df91bb46467d256e Oct 30 19:06:22.511240 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:22.511203 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 19:06:22.511335 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:22.511299 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 19:06:22.511379 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:22.511344 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 19:06:22.551197 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:22.551167 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-x95gb" Oct 30 19:06:22.551629 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:22.551561 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-x95gb" Oct 30 19:06:22.552479 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:22.552458 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-x95gb" Oct 30 19:06:22.687171 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:22.675715 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-cvflz" Oct 30 19:06:22.701446 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:22.691934 6960 patch_prober.go:29] interesting pod/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-cvflz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Oct 30 19:06:22.701446 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:22.692008 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-cvflz" podUID="eb8a7572-d545-404a-bb45-19afdbe8274d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Oct 30 19:06:22.790172 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:22.790029 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:23.225720 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:23.225677 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-xxv5m" event={"ID":"71138058-4eed-4e10-b831-50aa2d950f79","Type":"ContainerStarted","Data":"bdd65456188b4c11e5ee67c0131589a35888bea6e2be6ad3ed298ef02590579e"} Oct 30 19:06:23.225896 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:23.225734 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-xxv5m" event={"ID":"71138058-4eed-4e10-b831-50aa2d950f79","Type":"ContainerStarted","Data":"8a43e94145eef632458063af97b9a2e4b071e3ba9c19abd6df91bb46467d256e"} Oct 30 19:06:23.253379 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:23.253326 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-xxv5m" podStartSLOduration=2.253309334 podStartE2EDuration="2.253309334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:06:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:06:23.250295208 +0000 UTC m=+1950.779326345" watchObservedRunningTime="2026-10-30 19:06:23.253309334 +0000 UTC m=+1950.782340473" Oct 30 19:06:23.679582 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:23.679540 6960 patch_prober.go:29] interesting pod/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-cvflz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Oct 30 19:06:23.680095 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:23.679619 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-cvflz" podUID="eb8a7572-d545-404a-bb45-19afdbe8274d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Oct 30 19:06:23.781223 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:23.781185 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:23.974229 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:06:23.968990 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 19:06:24.150569 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:24.150538 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-xxv5m" Oct 30 19:06:24.152355 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:24.152329 6960 patch_prober.go:29] interesting pod/e2e-extmod-b-d72beb-openshift-default-869db9998f-xxv5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 30 19:06:24.152487 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:24.152391 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-xxv5m" podUID="71138058-4eed-4e10-b831-50aa2d950f79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 30 19:06:24.678172 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:24.672516 6960 patch_prober.go:29] interesting pod/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-cvflz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Oct 30 19:06:24.678172 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:24.672574 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-cvflz" podUID="eb8a7572-d545-404a-bb45-19afdbe8274d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Oct 30 19:06:24.813040 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:24.812628 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:25.150488 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:25.150443 6960 patch_prober.go:29] interesting pod/e2e-extmod-b-d72beb-openshift-default-869db9998f-xxv5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 30 19:06:25.150674 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:25.150513 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-xxv5m" podUID="71138058-4eed-4e10-b831-50aa2d950f79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 30 19:06:25.683170 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:25.677528 6960 patch_prober.go:29] interesting pod/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-cvflz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Oct 30 19:06:25.683170 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:25.677591 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-cvflz" podUID="eb8a7572-d545-404a-bb45-19afdbe8274d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Oct 30 19:06:26.127692 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:26.127646 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:26.150138 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:26.150105 6960 patch_prober.go:29] interesting pod/e2e-extmod-b-d72beb-openshift-default-869db9998f-xxv5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 30 19:06:26.150316 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:26.150184 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-xxv5m" podUID="71138058-4eed-4e10-b831-50aa2d950f79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 30 19:06:26.679170 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:26.678515 6960 patch_prober.go:29] interesting pod/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-cvflz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Oct 30 19:06:26.679170 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:26.678576 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-cvflz" podUID="eb8a7572-d545-404a-bb45-19afdbe8274d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Oct 30 19:06:26.901478 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:26.897638 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-c9e9de-7d8cd96456-wzsgn"] Oct 30 19:06:26.901478 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:26.897895 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c9e9de-7d8cd96456-wzsgn" podUID="6c861d5e-cd00-4c97-acda-9024e16bda48" containerName="maas-api" containerID="cri-o://e1a84b0975259af23efe36a6bb026487112896994ec66f24d00e13661b9b633d" gracePeriod=30 Oct 30 19:06:26.965185 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:26.963748 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:27.150417 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:27.150388 6960 patch_prober.go:29] interesting pod/e2e-extmod-b-d72beb-openshift-default-869db9998f-xxv5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 30 19:06:27.150892 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:27.150442 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-xxv5m" podUID="71138058-4eed-4e10-b831-50aa2d950f79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 30 19:06:27.344358 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:27.343352 6960 generic.go:356] "Generic (PLEG): container finished" podID="6c861d5e-cd00-4c97-acda-9024e16bda48" containerID="e1a84b0975259af23efe36a6bb026487112896994ec66f24d00e13661b9b633d" exitCode=0 Oct 30 19:06:27.344358 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:27.343559 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c9e9de-7d8cd96456-wzsgn" event={"ID":"6c861d5e-cd00-4c97-acda-9024e16bda48","Type":"ContainerDied","Data":"e1a84b0975259af23efe36a6bb026487112896994ec66f24d00e13661b9b633d"} Oct 30 19:06:27.466170 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:27.465775 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:27.645052 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:27.644762 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:27.683196 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:27.682667 6960 patch_prober.go:29] interesting pod/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-cvflz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Oct 30 19:06:27.683196 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:27.682719 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-cvflz" podUID="eb8a7572-d545-404a-bb45-19afdbe8274d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Oct 30 19:06:27.775573 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:27.774742 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:27.819850 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:27.819799 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c9e9de-7d8cd96456-wzsgn" Oct 30 19:06:27.840648 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:27.840613 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c861d5e-cd00-4c97-acda-9024e16bda48-kube-api-access-md7lt\" (UniqueName: \"kubernetes.io/projected/6c861d5e-cd00-4c97-acda-9024e16bda48-kube-api-access-md7lt\") pod \"6c861d5e-cd00-4c97-acda-9024e16bda48\" (UID: \"6c861d5e-cd00-4c97-acda-9024e16bda48\") " Oct 30 19:06:27.840823 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:27.840702 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c861d5e-cd00-4c97-acda-9024e16bda48-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c861d5e-cd00-4c97-acda-9024e16bda48-metrics-certs\") pod \"6c861d5e-cd00-4c97-acda-9024e16bda48\" (UID: \"6c861d5e-cd00-4c97-acda-9024e16bda48\") " Oct 30 19:06:27.840907 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:27.840834 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c861d5e-cd00-4c97-acda-9024e16bda48-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c861d5e-cd00-4c97-acda-9024e16bda48-openshift-service-ca\") pod \"6c861d5e-cd00-4c97-acda-9024e16bda48\" (UID: \"6c861d5e-cd00-4c97-acda-9024e16bda48\") " Oct 30 19:06:27.840971 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:27.840955 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c861d5e-cd00-4c97-acda-9024e16bda48-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c861d5e-cd00-4c97-acda-9024e16bda48-maas-api-tls\") pod \"6c861d5e-cd00-4c97-acda-9024e16bda48\" (UID: \"6c861d5e-cd00-4c97-acda-9024e16bda48\") " Oct 30 19:06:27.841204 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:27.841176 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c861d5e-cd00-4c97-acda-9024e16bda48-openshift-service-ca" pod "6c861d5e-cd00-4c97-acda-9024e16bda48" (UID: "6c861d5e-cd00-4c97-acda-9024e16bda48"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:06:27.841857 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:27.841830 6960 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c861d5e-cd00-4c97-acda-9024e16bda48-openshift-service-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:06:27.843786 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:27.843765 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c861d5e-cd00-4c97-acda-9024e16bda48-maas-api-tls" pod "6c861d5e-cd00-4c97-acda-9024e16bda48" (UID: "6c861d5e-cd00-4c97-acda-9024e16bda48"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:06:27.843864 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:27.843837 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c861d5e-cd00-4c97-acda-9024e16bda48-metrics-certs" pod "6c861d5e-cd00-4c97-acda-9024e16bda48" (UID: "6c861d5e-cd00-4c97-acda-9024e16bda48"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:06:27.902553 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:27.902384 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c861d5e-cd00-4c97-acda-9024e16bda48-kube-api-access-md7lt" pod "6c861d5e-cd00-4c97-acda-9024e16bda48" (UID: "6c861d5e-cd00-4c97-acda-9024e16bda48"). InnerVolumeSpecName "kube-api-access-md7lt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:06:27.968175 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:27.967819 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-md7lt\" (UniqueName: \"kubernetes.io/projected/6c861d5e-cd00-4c97-acda-9024e16bda48-kube-api-access-md7lt\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:06:27.968175 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:27.967854 6960 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c861d5e-cd00-4c97-acda-9024e16bda48-metrics-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:06:27.968175 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:27.967870 6960 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c861d5e-cd00-4c97-acda-9024e16bda48-maas-api-tls\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:06:28.150119 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:28.150077 6960 patch_prober.go:29] interesting pod/e2e-extmod-b-d72beb-openshift-default-869db9998f-xxv5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 30 19:06:28.150355 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:28.150188 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-xxv5m" podUID="71138058-4eed-4e10-b831-50aa2d950f79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 30 19:06:28.208451 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:28.208311 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:28.234281 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:28.234251 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7c6027-7b8d4f4fb9-mpf4x" Oct 30 19:06:28.392458 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:28.392403 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c9e9de-7d8cd96456-wzsgn" Oct 30 19:06:28.392644 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:28.392460 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c9e9de-7d8cd96456-wzsgn" event={"ID":"6c861d5e-cd00-4c97-acda-9024e16bda48","Type":"ContainerDied","Data":"c21abcf7234d595bf025a69f75ff2e822a846effba12098829b061e8cd9ffcf8"} Oct 30 19:06:28.392644 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:28.392527 6960 scope.go:122] "RemoveContainer" containerID="e1a84b0975259af23efe36a6bb026487112896994ec66f24d00e13661b9b633d" Oct 30 19:06:28.419469 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:28.419441 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-c9e9de-7d8cd96456-wzsgn"] Oct 30 19:06:28.423893 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:28.423871 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-c9e9de-7d8cd96456-wzsgn"] Oct 30 19:06:28.595579 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:28.595546 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c861d5e-cd00-4c97-acda-9024e16bda48" path="/var/lib/kubelet/pods/6c861d5e-cd00-4c97-acda-9024e16bda48/volumes" Oct 30 19:06:28.766170 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:28.765505 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-cvflz" Oct 30 19:06:28.775167 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:28.766997 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-cvflz" Oct 30 19:06:28.775660 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:28.775640 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-cvflz" Oct 30 19:06:29.095234 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:29.095194 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:29.150131 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:29.150097 6960 patch_prober.go:29] interesting pod/e2e-extmod-b-d72beb-openshift-default-869db9998f-xxv5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 30 19:06:29.150329 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:29.150180 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-xxv5m" podUID="71138058-4eed-4e10-b831-50aa2d950f79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 30 19:06:30.101766 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:30.101724 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:30.158183 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:30.158094 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-xxv5m" Oct 30 19:06:30.158531 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:30.158507 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-xxv5m" Oct 30 19:06:30.159850 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:30.159807 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-xxv5m" Oct 30 19:06:30.848985 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:30.848948 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:31.009349 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:31.009313 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:31.798382 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:31.798336 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:32.047246 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:32.047213 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:32.291353 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:32.291317 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:32.429013 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:32.428973 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:32.685778 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:32.685739 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:32.842358 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:32.842305 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:32.965222 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:32.964742 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:33.679762 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:33.677881 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:33.791992 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:33.791892 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:34.512207 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:34.512129 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:34.899095 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:34.899040 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:35.551931 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:35.551893 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:36.242604 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:36.242566 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:36.367251 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:36.367218 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:36.494882 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:36.494752 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:36.612344 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:36.612306 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-cvflz"] Oct 30 19:06:36.613514 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:36.613474 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-cvflz" podUID="eb8a7572-d545-404a-bb45-19afdbe8274d" containerName="istio-proxy" containerID="cri-o://c4e5966656ea077156dbdc2d0d060388d0eab8118b77c9e41accf0f16bb7a39a" gracePeriod=30 Oct 30 19:06:36.666179 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:36.666032 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:37.415286 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:37.415251 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:38.277578 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:38.277536 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:38.546360 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:38.546257 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:39.094763 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:39.094725 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:39.218822 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:39.218785 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:39.613306 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:39.613277 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:40.511492 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:40.511457 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:41.697568 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:41.697527 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:41.952765 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:41.952740 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-cvflz" Oct 30 19:06:42.080027 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:42.079995 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb8a7572-d545-404a-bb45-19afdbe8274d-istio-token\" (UniqueName: \"kubernetes.io/projected/eb8a7572-d545-404a-bb45-19afdbe8274d-istio-token\") pod \"eb8a7572-d545-404a-bb45-19afdbe8274d\" (UID: \"eb8a7572-d545-404a-bb45-19afdbe8274d\") " Oct 30 19:06:42.080233 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:42.080056 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb8a7572-d545-404a-bb45-19afdbe8274d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb8a7572-d545-404a-bb45-19afdbe8274d-workload-certs\") pod \"eb8a7572-d545-404a-bb45-19afdbe8274d\" (UID: \"eb8a7572-d545-404a-bb45-19afdbe8274d\") " Oct 30 19:06:42.080286 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:42.080230 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb8a7572-d545-404a-bb45-19afdbe8274d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb8a7572-d545-404a-bb45-19afdbe8274d-workload-socket\") pod \"eb8a7572-d545-404a-bb45-19afdbe8274d\" (UID: \"eb8a7572-d545-404a-bb45-19afdbe8274d\") " Oct 30 19:06:42.080339 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:42.080319 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb8a7572-d545-404a-bb45-19afdbe8274d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb8a7572-d545-404a-bb45-19afdbe8274d-istio-data\") pod \"eb8a7572-d545-404a-bb45-19afdbe8274d\" (UID: \"eb8a7572-d545-404a-bb45-19afdbe8274d\") " Oct 30 19:06:42.080383 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:42.080351 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb8a7572-d545-404a-bb45-19afdbe8274d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb8a7572-d545-404a-bb45-19afdbe8274d-credential-socket\") pod \"eb8a7572-d545-404a-bb45-19afdbe8274d\" (UID: \"eb8a7572-d545-404a-bb45-19afdbe8274d\") " Oct 30 19:06:42.080438 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:42.080370 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb8a7572-d545-404a-bb45-19afdbe8274d-workload-certs" pod "eb8a7572-d545-404a-bb45-19afdbe8274d" (UID: "eb8a7572-d545-404a-bb45-19afdbe8274d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:06:42.080438 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:42.080385 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb8a7572-d545-404a-bb45-19afdbe8274d-kube-api-access-tm6k4\" (UniqueName: \"kubernetes.io/projected/eb8a7572-d545-404a-bb45-19afdbe8274d-kube-api-access-tm6k4\") pod \"eb8a7572-d545-404a-bb45-19afdbe8274d\" (UID: \"eb8a7572-d545-404a-bb45-19afdbe8274d\") " Oct 30 19:06:42.080554 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:42.080447 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb8a7572-d545-404a-bb45-19afdbe8274d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb8a7572-d545-404a-bb45-19afdbe8274d-istiod-ca-cert\") pod \"eb8a7572-d545-404a-bb45-19afdbe8274d\" (UID: \"eb8a7572-d545-404a-bb45-19afdbe8274d\") " Oct 30 19:06:42.080554 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:42.080478 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb8a7572-d545-404a-bb45-19afdbe8274d-workload-socket" pod "eb8a7572-d545-404a-bb45-19afdbe8274d" (UID: "eb8a7572-d545-404a-bb45-19afdbe8274d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:06:42.080706 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:42.080574 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb8a7572-d545-404a-bb45-19afdbe8274d-credential-socket" pod "eb8a7572-d545-404a-bb45-19afdbe8274d" (UID: "eb8a7572-d545-404a-bb45-19afdbe8274d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:06:42.080706 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:42.080594 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb8a7572-d545-404a-bb45-19afdbe8274d-istio-data" pod "eb8a7572-d545-404a-bb45-19afdbe8274d" (UID: "eb8a7572-d545-404a-bb45-19afdbe8274d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:06:42.080706 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:42.080616 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eb8a7572-d545-404a-bb45-19afdbe8274d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb8a7572-d545-404a-bb45-19afdbe8274d-istio-podinfo\") pod \"eb8a7572-d545-404a-bb45-19afdbe8274d\" (UID: \"eb8a7572-d545-404a-bb45-19afdbe8274d\") " Oct 30 19:06:42.080901 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:42.080817 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb8a7572-d545-404a-bb45-19afdbe8274d-istiod-ca-cert" pod "eb8a7572-d545-404a-bb45-19afdbe8274d" (UID: "eb8a7572-d545-404a-bb45-19afdbe8274d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:06:42.080901 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:42.080807 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb8a7572-d545-404a-bb45-19afdbe8274d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb8a7572-d545-404a-bb45-19afdbe8274d-istio-envoy\") pod \"eb8a7572-d545-404a-bb45-19afdbe8274d\" (UID: \"eb8a7572-d545-404a-bb45-19afdbe8274d\") " Oct 30 19:06:42.081651 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:42.081598 6960 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb8a7572-d545-404a-bb45-19afdbe8274d-workload-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:06:42.081651 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:42.081626 6960 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb8a7572-d545-404a-bb45-19afdbe8274d-workload-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:06:42.081651 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:42.081641 6960 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb8a7572-d545-404a-bb45-19afdbe8274d-istio-data\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:06:42.081875 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:42.081654 6960 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb8a7572-d545-404a-bb45-19afdbe8274d-credential-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:06:42.081875 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:42.081670 6960 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb8a7572-d545-404a-bb45-19afdbe8274d-istiod-ca-cert\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:06:42.082677 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:42.082644 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb8a7572-d545-404a-bb45-19afdbe8274d-istio-token" pod "eb8a7572-d545-404a-bb45-19afdbe8274d" (UID: "eb8a7572-d545-404a-bb45-19afdbe8274d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:06:42.082778 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:42.082681 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eb8a7572-d545-404a-bb45-19afdbe8274d-istio-podinfo" pod "eb8a7572-d545-404a-bb45-19afdbe8274d" (UID: "eb8a7572-d545-404a-bb45-19afdbe8274d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:06:42.082823 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:42.082767 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb8a7572-d545-404a-bb45-19afdbe8274d-kube-api-access-tm6k4" pod "eb8a7572-d545-404a-bb45-19afdbe8274d" (UID: "eb8a7572-d545-404a-bb45-19afdbe8274d"). InnerVolumeSpecName "kube-api-access-tm6k4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:06:42.083013 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:42.082998 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb8a7572-d545-404a-bb45-19afdbe8274d-istio-envoy" pod "eb8a7572-d545-404a-bb45-19afdbe8274d" (UID: "eb8a7572-d545-404a-bb45-19afdbe8274d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:06:42.182845 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:42.182776 6960 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb8a7572-d545-404a-bb45-19afdbe8274d-istio-envoy\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:06:42.182845 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:42.182803 6960 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb8a7572-d545-404a-bb45-19afdbe8274d-istio-token\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:06:42.182845 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:42.182814 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tm6k4\" (UniqueName: \"kubernetes.io/projected/eb8a7572-d545-404a-bb45-19afdbe8274d-kube-api-access-tm6k4\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:06:42.182845 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:42.182824 6960 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb8a7572-d545-404a-bb45-19afdbe8274d-istio-podinfo\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:06:42.546012 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:42.545924 6960 generic.go:356] "Generic (PLEG): container finished" podID="eb8a7572-d545-404a-bb45-19afdbe8274d" containerID="c4e5966656ea077156dbdc2d0d060388d0eab8118b77c9e41accf0f16bb7a39a" exitCode=0 Oct 30 19:06:42.546012 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:42.545968 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-cvflz" Oct 30 19:06:42.546279 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:42.546012 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-cvflz" event={"ID":"eb8a7572-d545-404a-bb45-19afdbe8274d","Type":"ContainerDied","Data":"c4e5966656ea077156dbdc2d0d060388d0eab8118b77c9e41accf0f16bb7a39a"} Oct 30 19:06:42.546279 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:42.546063 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-cvflz" event={"ID":"eb8a7572-d545-404a-bb45-19afdbe8274d","Type":"ContainerDied","Data":"bd58db3a469d6b474e373f718c06e3c3758ce6b722a084c4e1c1b1a50b2194f9"} Oct 30 19:06:42.546279 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:42.546096 6960 scope.go:122] "RemoveContainer" containerID="c4e5966656ea077156dbdc2d0d060388d0eab8118b77c9e41accf0f16bb7a39a" Oct 30 19:06:42.558521 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:42.558502 6960 scope.go:122] "RemoveContainer" containerID="c4e5966656ea077156dbdc2d0d060388d0eab8118b77c9e41accf0f16bb7a39a" Oct 30 19:06:42.558799 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:06:42.558774 6960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4e5966656ea077156dbdc2d0d060388d0eab8118b77c9e41accf0f16bb7a39a\": container with ID starting with c4e5966656ea077156dbdc2d0d060388d0eab8118b77c9e41accf0f16bb7a39a not found: ID does not exist" containerID="c4e5966656ea077156dbdc2d0d060388d0eab8118b77c9e41accf0f16bb7a39a" Oct 30 19:06:42.558886 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:42.558810 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4e5966656ea077156dbdc2d0d060388d0eab8118b77c9e41accf0f16bb7a39a"} err="failed to get container status \"c4e5966656ea077156dbdc2d0d060388d0eab8118b77c9e41accf0f16bb7a39a\": rpc error: code = NotFound desc = could not find container \"c4e5966656ea077156dbdc2d0d060388d0eab8118b77c9e41accf0f16bb7a39a\": container with ID starting with c4e5966656ea077156dbdc2d0d060388d0eab8118b77c9e41accf0f16bb7a39a not found: ID does not exist" Oct 30 19:06:42.579973 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:42.579926 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-cvflz"] Oct 30 19:06:42.583305 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:42.583280 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-c9e9de-openshift-default-7fcd5c9d7c-cvflz"] Oct 30 19:06:42.596316 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:42.596290 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb8a7572-d545-404a-bb45-19afdbe8274d" path="/var/lib/kubelet/pods/eb8a7572-d545-404a-bb45-19afdbe8274d/volumes" Oct 30 19:06:42.639029 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:42.638992 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:43.498932 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:43.498901 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:44.044462 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:44.044415 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:44.186950 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:44.186912 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:44.722827 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:44.722781 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:45.329559 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:45.329524 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:45.539337 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:45.539292 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:46.024041 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:46.024004 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:47.104800 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:47.104766 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:47.872077 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:47.872034 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:48.354110 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:48.354050 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:49.139611 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:49.139575 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:49.261027 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:49.260996 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:49.380472 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:49.380442 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:49.517374 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:49.517283 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:49.767848 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:49.767762 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:49.937984 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:49.937949 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:50.065603 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:50.065573 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:50.321815 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:50.321722 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:50.534901 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:50.534853 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:50.668916 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:50.668823 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:51.515671 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:51.515638 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:51.863477 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:51.863439 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:52.449835 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:52.449800 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:52.927192 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:52.927123 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:53.434646 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:53.434619 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:53.933360 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:06:53.933323 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 19:06:54.290862 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:54.290753 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:55.206226 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:55.206187 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:55.985543 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:55.985512 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:57.009723 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:57.009690 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:57.951631 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:57.951592 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:58.077263 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:58.077227 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:58.807437 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:58.807395 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:06:59.931894 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:06:59.931855 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:00.318658 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:00.318631 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-qhvj4"] Oct 30 19:07:00.319461 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:00.319439 6960 cpu_manager.go:414] "containerMap: removing container" podUID="eb8a7572-d545-404a-bb45-19afdbe8274d" containerName="istio-proxy" Oct 30 19:07:00.319461 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:00.319459 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb8a7572-d545-404a-bb45-19afdbe8274d" containerName="istio-proxy" Oct 30 19:07:00.319641 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:00.319485 6960 cpu_manager.go:414] "containerMap: removing container" podUID="6c861d5e-cd00-4c97-acda-9024e16bda48" containerName="maas-api" Oct 30 19:07:00.319641 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:00.319490 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c861d5e-cd00-4c97-acda-9024e16bda48" containerName="maas-api" Oct 30 19:07:00.319641 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:00.319620 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb8a7572-d545-404a-bb45-19afdbe8274d" containerName="istio-proxy" Oct 30 19:07:00.319641 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:00.319633 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c861d5e-cd00-4c97-acda-9024e16bda48" containerName="maas-api" Oct 30 19:07:00.325326 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:00.325303 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-qhvj4" Oct 30 19:07:00.348935 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:00.348898 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-qhvj4"] Oct 30 19:07:00.474477 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:00.474440 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b06be103-9039-4229-bb04-d464932e988f-istio-podinfo\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-qhvj4\" (UID: \"b06be103-9039-4229-bb04-d464932e988f\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-qhvj4" Oct 30 19:07:00.474664 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:00.474485 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b06be103-9039-4229-bb04-d464932e988f-workload-socket\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-qhvj4\" (UID: \"b06be103-9039-4229-bb04-d464932e988f\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-qhvj4" Oct 30 19:07:00.474664 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:00.474558 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b06be103-9039-4229-bb04-d464932e988f-istio-envoy\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-qhvj4\" (UID: \"b06be103-9039-4229-bb04-d464932e988f\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-qhvj4" Oct 30 19:07:00.474664 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:00.474641 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b06be103-9039-4229-bb04-d464932e988f-istio-token\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-qhvj4\" (UID: \"b06be103-9039-4229-bb04-d464932e988f\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-qhvj4" Oct 30 19:07:00.474909 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:00.474819 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6t94\" (UniqueName: \"kubernetes.io/projected/b06be103-9039-4229-bb04-d464932e988f-kube-api-access-b6t94\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-qhvj4\" (UID: \"b06be103-9039-4229-bb04-d464932e988f\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-qhvj4" Oct 30 19:07:00.474909 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:00.474863 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b06be103-9039-4229-bb04-d464932e988f-istiod-ca-cert\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-qhvj4\" (UID: \"b06be103-9039-4229-bb04-d464932e988f\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-qhvj4" Oct 30 19:07:00.475023 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:00.474963 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b06be103-9039-4229-bb04-d464932e988f-workload-certs\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-qhvj4\" (UID: \"b06be103-9039-4229-bb04-d464932e988f\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-qhvj4" Oct 30 19:07:00.475080 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:00.475042 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b06be103-9039-4229-bb04-d464932e988f-istio-data\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-qhvj4\" (UID: \"b06be103-9039-4229-bb04-d464932e988f\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-qhvj4" Oct 30 19:07:00.475080 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:00.475069 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b06be103-9039-4229-bb04-d464932e988f-credential-socket\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-qhvj4\" (UID: \"b06be103-9039-4229-bb04-d464932e988f\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-qhvj4" Oct 30 19:07:00.576983 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:00.576909 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b06be103-9039-4229-bb04-d464932e988f-istio-podinfo\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-qhvj4\" (UID: \"b06be103-9039-4229-bb04-d464932e988f\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-qhvj4" Oct 30 19:07:00.576983 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:00.576942 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b06be103-9039-4229-bb04-d464932e988f-workload-socket\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-qhvj4\" (UID: \"b06be103-9039-4229-bb04-d464932e988f\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-qhvj4" Oct 30 19:07:00.576983 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:00.576963 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b06be103-9039-4229-bb04-d464932e988f-istio-envoy\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-qhvj4\" (UID: \"b06be103-9039-4229-bb04-d464932e988f\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-qhvj4" Oct 30 19:07:00.577328 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:00.577094 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b06be103-9039-4229-bb04-d464932e988f-istio-token\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-qhvj4\" (UID: \"b06be103-9039-4229-bb04-d464932e988f\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-qhvj4" Oct 30 19:07:00.577390 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:00.577347 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6t94\" (UniqueName: \"kubernetes.io/projected/b06be103-9039-4229-bb04-d464932e988f-kube-api-access-b6t94\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-qhvj4\" (UID: \"b06be103-9039-4229-bb04-d464932e988f\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-qhvj4" Oct 30 19:07:00.577443 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:00.577397 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b06be103-9039-4229-bb04-d464932e988f-workload-socket\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-qhvj4\" (UID: \"b06be103-9039-4229-bb04-d464932e988f\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-qhvj4" Oct 30 19:07:00.577443 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:00.577411 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b06be103-9039-4229-bb04-d464932e988f-istiod-ca-cert\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-qhvj4\" (UID: \"b06be103-9039-4229-bb04-d464932e988f\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-qhvj4" Oct 30 19:07:00.577547 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:00.577456 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b06be103-9039-4229-bb04-d464932e988f-workload-certs\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-qhvj4\" (UID: \"b06be103-9039-4229-bb04-d464932e988f\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-qhvj4" Oct 30 19:07:00.577610 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:00.577547 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b06be103-9039-4229-bb04-d464932e988f-istio-data\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-qhvj4\" (UID: \"b06be103-9039-4229-bb04-d464932e988f\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-qhvj4" Oct 30 19:07:00.577610 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:00.577576 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b06be103-9039-4229-bb04-d464932e988f-credential-socket\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-qhvj4\" (UID: \"b06be103-9039-4229-bb04-d464932e988f\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-qhvj4" Oct 30 19:07:00.577768 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:00.577738 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b06be103-9039-4229-bb04-d464932e988f-workload-certs\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-qhvj4\" (UID: \"b06be103-9039-4229-bb04-d464932e988f\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-qhvj4" Oct 30 19:07:00.577932 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:00.577909 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b06be103-9039-4229-bb04-d464932e988f-credential-socket\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-qhvj4\" (UID: \"b06be103-9039-4229-bb04-d464932e988f\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-qhvj4" Oct 30 19:07:00.578073 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:00.578056 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b06be103-9039-4229-bb04-d464932e988f-istiod-ca-cert\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-qhvj4\" (UID: \"b06be103-9039-4229-bb04-d464932e988f\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-qhvj4" Oct 30 19:07:00.578130 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:00.578059 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b06be103-9039-4229-bb04-d464932e988f-istio-data\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-qhvj4\" (UID: \"b06be103-9039-4229-bb04-d464932e988f\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-qhvj4" Oct 30 19:07:00.579604 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:00.579582 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b06be103-9039-4229-bb04-d464932e988f-istio-envoy\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-qhvj4\" (UID: \"b06be103-9039-4229-bb04-d464932e988f\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-qhvj4" Oct 30 19:07:00.579738 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:00.579721 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b06be103-9039-4229-bb04-d464932e988f-istio-podinfo\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-qhvj4\" (UID: \"b06be103-9039-4229-bb04-d464932e988f\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-qhvj4" Oct 30 19:07:00.589108 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:00.589089 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6t94\" (UniqueName: \"kubernetes.io/projected/b06be103-9039-4229-bb04-d464932e988f-kube-api-access-b6t94\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-qhvj4\" (UID: \"b06be103-9039-4229-bb04-d464932e988f\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-qhvj4" Oct 30 19:07:00.592703 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:00.592679 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b06be103-9039-4229-bb04-d464932e988f-istio-token\") pod \"e2e-extmod-a-de4bda-openshift-default-5766cd8c57-qhvj4\" (UID: \"b06be103-9039-4229-bb04-d464932e988f\") " pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-qhvj4" Oct 30 19:07:00.651833 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:00.651797 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-qhvj4" Oct 30 19:07:00.939599 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:00.939510 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:00.958552 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:00.958520 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-qhvj4"] Oct 30 19:07:00.986357 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:00.985343 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 19:07:00.986610 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:00.986582 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 19:07:00.986717 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:00.986640 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 19:07:01.787789 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:01.787746 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-qhvj4" event={"ID":"b06be103-9039-4229-bb04-d464932e988f","Type":"ContainerStarted","Data":"fcbb28957b6101db73040aae408c3745b8d1854c2fcb157b2522b30ed6ddf45b"} Oct 30 19:07:01.787980 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:01.787806 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-qhvj4" event={"ID":"b06be103-9039-4229-bb04-d464932e988f","Type":"ContainerStarted","Data":"264b0054f2853d993e9d94b8c86a8083a75a97063f6ae4be975b6a5f2db95dbf"} Oct 30 19:07:01.813313 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:01.813253 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-qhvj4" podStartSLOduration=1.813235636 podStartE2EDuration="1.813235636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:07:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:07:01.810508989 +0000 UTC m=+1989.339540130" watchObservedRunningTime="2026-10-30 19:07:01.813235636 +0000 UTC m=+1989.342266775" Oct 30 19:07:01.949608 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:01.949571 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:02.652646 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:02.652609 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-qhvj4" Oct 30 19:07:02.679171 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:02.675617 6960 patch_prober.go:29] interesting pod/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-qhvj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 19:07:02.679171 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:02.675675 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-qhvj4" podUID="b06be103-9039-4229-bb04-d464932e988f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 19:07:03.042570 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:03.042470 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:03.652588 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:03.652502 6960 patch_prober.go:29] interesting pod/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-qhvj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 19:07:03.652588 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:03.652568 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-qhvj4" podUID="b06be103-9039-4229-bb04-d464932e988f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 19:07:04.018980 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:04.018868 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:04.652059 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:04.652019 6960 patch_prober.go:29] interesting pod/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-qhvj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 19:07:04.652473 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:04.652074 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-qhvj4" podUID="b06be103-9039-4229-bb04-d464932e988f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 19:07:04.926284 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:04.926191 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:05.652136 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:05.652101 6960 patch_prober.go:29] interesting pod/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-qhvj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 19:07:05.652579 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:05.652187 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-qhvj4" podUID="b06be103-9039-4229-bb04-d464932e988f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 19:07:05.811753 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:05.811718 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:06.611326 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:06.611287 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-pzmg6"] Oct 30 19:07:06.619418 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:06.619390 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-pzmg6" Oct 30 19:07:06.625758 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:06.625687 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-7c6027-openshift-default-dockercfg-7c5ll\"" Oct 30 19:07:06.635280 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:06.635252 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-pzmg6"] Oct 30 19:07:06.652400 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:06.652360 6960 patch_prober.go:29] interesting pod/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-qhvj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 19:07:06.652813 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:06.652419 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-qhvj4" podUID="b06be103-9039-4229-bb04-d464932e988f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 19:07:06.694464 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:06.694431 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrpcg\" (UniqueName: \"kubernetes.io/projected/50c12729-1387-436a-bee0-9eb3316db708-kube-api-access-vrpcg\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-pzmg6\" (UID: \"50c12729-1387-436a-bee0-9eb3316db708\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-pzmg6" Oct 30 19:07:06.694643 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:06.694481 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50c12729-1387-436a-bee0-9eb3316db708-istio-envoy\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-pzmg6\" (UID: \"50c12729-1387-436a-bee0-9eb3316db708\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-pzmg6" Oct 30 19:07:06.694643 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:06.694602 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50c12729-1387-436a-bee0-9eb3316db708-istio-token\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-pzmg6\" (UID: \"50c12729-1387-436a-bee0-9eb3316db708\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-pzmg6" Oct 30 19:07:06.694769 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:06.694654 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50c12729-1387-436a-bee0-9eb3316db708-istio-data\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-pzmg6\" (UID: \"50c12729-1387-436a-bee0-9eb3316db708\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-pzmg6" Oct 30 19:07:06.694769 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:06.694688 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50c12729-1387-436a-bee0-9eb3316db708-workload-socket\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-pzmg6\" (UID: \"50c12729-1387-436a-bee0-9eb3316db708\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-pzmg6" Oct 30 19:07:06.694983 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:06.694963 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50c12729-1387-436a-bee0-9eb3316db708-credential-socket\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-pzmg6\" (UID: \"50c12729-1387-436a-bee0-9eb3316db708\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-pzmg6" Oct 30 19:07:06.695079 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:06.695064 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50c12729-1387-436a-bee0-9eb3316db708-istio-podinfo\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-pzmg6\" (UID: \"50c12729-1387-436a-bee0-9eb3316db708\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-pzmg6" Oct 30 19:07:06.695231 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:06.695137 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50c12729-1387-436a-bee0-9eb3316db708-workload-certs\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-pzmg6\" (UID: \"50c12729-1387-436a-bee0-9eb3316db708\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-pzmg6" Oct 30 19:07:06.695395 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:06.695378 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50c12729-1387-436a-bee0-9eb3316db708-istiod-ca-cert\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-pzmg6\" (UID: \"50c12729-1387-436a-bee0-9eb3316db708\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-pzmg6" Oct 30 19:07:06.780929 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:06.780881 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:06.801941 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:06.801652 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50c12729-1387-436a-bee0-9eb3316db708-workload-socket\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-pzmg6\" (UID: \"50c12729-1387-436a-bee0-9eb3316db708\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-pzmg6" Oct 30 19:07:06.801941 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:06.801762 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50c12729-1387-436a-bee0-9eb3316db708-credential-socket\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-pzmg6\" (UID: \"50c12729-1387-436a-bee0-9eb3316db708\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-pzmg6" Oct 30 19:07:06.801941 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:06.801802 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50c12729-1387-436a-bee0-9eb3316db708-istio-podinfo\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-pzmg6\" (UID: \"50c12729-1387-436a-bee0-9eb3316db708\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-pzmg6" Oct 30 19:07:06.801941 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:06.801840 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50c12729-1387-436a-bee0-9eb3316db708-workload-certs\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-pzmg6\" (UID: \"50c12729-1387-436a-bee0-9eb3316db708\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-pzmg6" Oct 30 19:07:06.802355 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:06.801969 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50c12729-1387-436a-bee0-9eb3316db708-istiod-ca-cert\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-pzmg6\" (UID: \"50c12729-1387-436a-bee0-9eb3316db708\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-pzmg6" Oct 30 19:07:06.802355 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:06.802129 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrpcg\" (UniqueName: \"kubernetes.io/projected/50c12729-1387-436a-bee0-9eb3316db708-kube-api-access-vrpcg\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-pzmg6\" (UID: \"50c12729-1387-436a-bee0-9eb3316db708\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-pzmg6" Oct 30 19:07:06.802355 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:06.802185 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50c12729-1387-436a-bee0-9eb3316db708-istio-envoy\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-pzmg6\" (UID: \"50c12729-1387-436a-bee0-9eb3316db708\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-pzmg6" Oct 30 19:07:06.802355 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:06.802238 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50c12729-1387-436a-bee0-9eb3316db708-istio-token\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-pzmg6\" (UID: \"50c12729-1387-436a-bee0-9eb3316db708\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-pzmg6" Oct 30 19:07:06.802355 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:06.802284 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50c12729-1387-436a-bee0-9eb3316db708-istio-data\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-pzmg6\" (UID: \"50c12729-1387-436a-bee0-9eb3316db708\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-pzmg6" Oct 30 19:07:06.803190 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:06.803000 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50c12729-1387-436a-bee0-9eb3316db708-workload-certs\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-pzmg6\" (UID: \"50c12729-1387-436a-bee0-9eb3316db708\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-pzmg6" Oct 30 19:07:06.803190 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:06.803030 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50c12729-1387-436a-bee0-9eb3316db708-workload-socket\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-pzmg6\" (UID: \"50c12729-1387-436a-bee0-9eb3316db708\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-pzmg6" Oct 30 19:07:06.803387 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:06.803298 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50c12729-1387-436a-bee0-9eb3316db708-istio-data\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-pzmg6\" (UID: \"50c12729-1387-436a-bee0-9eb3316db708\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-pzmg6" Oct 30 19:07:06.803609 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:06.803587 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50c12729-1387-436a-bee0-9eb3316db708-credential-socket\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-pzmg6\" (UID: \"50c12729-1387-436a-bee0-9eb3316db708\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-pzmg6" Oct 30 19:07:06.805340 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:06.804243 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50c12729-1387-436a-bee0-9eb3316db708-istiod-ca-cert\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-pzmg6\" (UID: \"50c12729-1387-436a-bee0-9eb3316db708\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-pzmg6" Oct 30 19:07:06.807103 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:06.807075 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50c12729-1387-436a-bee0-9eb3316db708-istio-podinfo\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-pzmg6\" (UID: \"50c12729-1387-436a-bee0-9eb3316db708\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-pzmg6" Oct 30 19:07:06.807497 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:06.807189 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50c12729-1387-436a-bee0-9eb3316db708-istio-envoy\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-pzmg6\" (UID: \"50c12729-1387-436a-bee0-9eb3316db708\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-pzmg6" Oct 30 19:07:06.818168 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:06.818024 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrpcg\" (UniqueName: \"kubernetes.io/projected/50c12729-1387-436a-bee0-9eb3316db708-kube-api-access-vrpcg\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-pzmg6\" (UID: \"50c12729-1387-436a-bee0-9eb3316db708\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-pzmg6" Oct 30 19:07:06.820476 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:06.820456 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50c12729-1387-436a-bee0-9eb3316db708-istio-token\") pod \"e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-pzmg6\" (UID: \"50c12729-1387-436a-bee0-9eb3316db708\") " pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-pzmg6" Oct 30 19:07:06.941944 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:06.941865 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-pzmg6" Oct 30 19:07:06.995678 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:06.995373 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-wfw84"] Oct 30 19:07:07.004467 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:07.004441 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-wfw84"] Oct 30 19:07:07.004751 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:07.004725 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-wfw84" Oct 30 19:07:07.108439 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:07.108401 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da02a9d7-7b75-401c-9134-8e45d7303175-istio-data\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-wfw84\" (UID: \"da02a9d7-7b75-401c-9134-8e45d7303175\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-wfw84" Oct 30 19:07:07.108616 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:07.108521 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da02a9d7-7b75-401c-9134-8e45d7303175-workload-certs\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-wfw84\" (UID: \"da02a9d7-7b75-401c-9134-8e45d7303175\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-wfw84" Oct 30 19:07:07.108616 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:07.108601 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da02a9d7-7b75-401c-9134-8e45d7303175-istio-token\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-wfw84\" (UID: \"da02a9d7-7b75-401c-9134-8e45d7303175\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-wfw84" Oct 30 19:07:07.108773 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:07.108754 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da02a9d7-7b75-401c-9134-8e45d7303175-istiod-ca-cert\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-wfw84\" (UID: \"da02a9d7-7b75-401c-9134-8e45d7303175\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-wfw84" Oct 30 19:07:07.108836 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:07.108793 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da02a9d7-7b75-401c-9134-8e45d7303175-istio-podinfo\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-wfw84\" (UID: \"da02a9d7-7b75-401c-9134-8e45d7303175\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-wfw84" Oct 30 19:07:07.108836 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:07.108827 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da02a9d7-7b75-401c-9134-8e45d7303175-istio-envoy\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-wfw84\" (UID: \"da02a9d7-7b75-401c-9134-8e45d7303175\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-wfw84" Oct 30 19:07:07.108936 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:07.108908 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da02a9d7-7b75-401c-9134-8e45d7303175-workload-socket\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-wfw84\" (UID: \"da02a9d7-7b75-401c-9134-8e45d7303175\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-wfw84" Oct 30 19:07:07.108989 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:07.108943 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85wvp\" (UniqueName: \"kubernetes.io/projected/da02a9d7-7b75-401c-9134-8e45d7303175-kube-api-access-85wvp\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-wfw84\" (UID: \"da02a9d7-7b75-401c-9134-8e45d7303175\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-wfw84" Oct 30 19:07:07.109064 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:07.109051 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da02a9d7-7b75-401c-9134-8e45d7303175-credential-socket\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-wfw84\" (UID: \"da02a9d7-7b75-401c-9134-8e45d7303175\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-wfw84" Oct 30 19:07:07.211502 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:07.211401 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da02a9d7-7b75-401c-9134-8e45d7303175-istiod-ca-cert\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-wfw84\" (UID: \"da02a9d7-7b75-401c-9134-8e45d7303175\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-wfw84" Oct 30 19:07:07.211502 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:07.211458 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da02a9d7-7b75-401c-9134-8e45d7303175-istio-podinfo\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-wfw84\" (UID: \"da02a9d7-7b75-401c-9134-8e45d7303175\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-wfw84" Oct 30 19:07:07.211733 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:07.211502 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da02a9d7-7b75-401c-9134-8e45d7303175-istio-envoy\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-wfw84\" (UID: \"da02a9d7-7b75-401c-9134-8e45d7303175\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-wfw84" Oct 30 19:07:07.211733 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:07.211583 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da02a9d7-7b75-401c-9134-8e45d7303175-workload-socket\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-wfw84\" (UID: \"da02a9d7-7b75-401c-9134-8e45d7303175\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-wfw84" Oct 30 19:07:07.211733 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:07.211665 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85wvp\" (UniqueName: \"kubernetes.io/projected/da02a9d7-7b75-401c-9134-8e45d7303175-kube-api-access-85wvp\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-wfw84\" (UID: \"da02a9d7-7b75-401c-9134-8e45d7303175\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-wfw84" Oct 30 19:07:07.211858 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:07.211770 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da02a9d7-7b75-401c-9134-8e45d7303175-credential-socket\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-wfw84\" (UID: \"da02a9d7-7b75-401c-9134-8e45d7303175\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-wfw84" Oct 30 19:07:07.211858 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:07.211849 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da02a9d7-7b75-401c-9134-8e45d7303175-istio-data\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-wfw84\" (UID: \"da02a9d7-7b75-401c-9134-8e45d7303175\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-wfw84" Oct 30 19:07:07.211923 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:07.211907 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da02a9d7-7b75-401c-9134-8e45d7303175-workload-certs\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-wfw84\" (UID: \"da02a9d7-7b75-401c-9134-8e45d7303175\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-wfw84" Oct 30 19:07:07.211984 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:07.211965 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da02a9d7-7b75-401c-9134-8e45d7303175-istio-token\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-wfw84\" (UID: \"da02a9d7-7b75-401c-9134-8e45d7303175\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-wfw84" Oct 30 19:07:07.212257 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:07.212229 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da02a9d7-7b75-401c-9134-8e45d7303175-istiod-ca-cert\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-wfw84\" (UID: \"da02a9d7-7b75-401c-9134-8e45d7303175\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-wfw84" Oct 30 19:07:07.212640 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:07.212508 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da02a9d7-7b75-401c-9134-8e45d7303175-istio-data\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-wfw84\" (UID: \"da02a9d7-7b75-401c-9134-8e45d7303175\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-wfw84" Oct 30 19:07:07.212735 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:07.212718 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da02a9d7-7b75-401c-9134-8e45d7303175-workload-certs\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-wfw84\" (UID: \"da02a9d7-7b75-401c-9134-8e45d7303175\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-wfw84" Oct 30 19:07:07.212850 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:07.212824 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da02a9d7-7b75-401c-9134-8e45d7303175-credential-socket\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-wfw84\" (UID: \"da02a9d7-7b75-401c-9134-8e45d7303175\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-wfw84" Oct 30 19:07:07.212921 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:07.212837 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da02a9d7-7b75-401c-9134-8e45d7303175-workload-socket\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-wfw84\" (UID: \"da02a9d7-7b75-401c-9134-8e45d7303175\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-wfw84" Oct 30 19:07:07.214105 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:07.214080 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da02a9d7-7b75-401c-9134-8e45d7303175-istio-envoy\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-wfw84\" (UID: \"da02a9d7-7b75-401c-9134-8e45d7303175\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-wfw84" Oct 30 19:07:07.214658 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:07.214641 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da02a9d7-7b75-401c-9134-8e45d7303175-istio-podinfo\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-wfw84\" (UID: \"da02a9d7-7b75-401c-9134-8e45d7303175\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-wfw84" Oct 30 19:07:07.224213 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:07.224184 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85wvp\" (UniqueName: \"kubernetes.io/projected/da02a9d7-7b75-401c-9134-8e45d7303175-kube-api-access-85wvp\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-wfw84\" (UID: \"da02a9d7-7b75-401c-9134-8e45d7303175\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-wfw84" Oct 30 19:07:07.224461 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:07.224437 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da02a9d7-7b75-401c-9134-8e45d7303175-istio-token\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-wfw84\" (UID: \"da02a9d7-7b75-401c-9134-8e45d7303175\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-wfw84" Oct 30 19:07:07.279569 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:07.279542 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-pzmg6"] Oct 30 19:07:07.280590 ip-10-0-140-163 kubenswrapper[6960]: W1030 19:07:07.280562 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50c12729_1387_436a_bee0_9eb3316db708.slice/crio-b8887ac9dbd301fd7badfb0721c9ed2d6dc611b2b7ccf7d05aab7c4b823f5146 WatchSource:0}: Error finding container b8887ac9dbd301fd7badfb0721c9ed2d6dc611b2b7ccf7d05aab7c4b823f5146: Status 404 returned error can't find the container with id b8887ac9dbd301fd7badfb0721c9ed2d6dc611b2b7ccf7d05aab7c4b823f5146 Oct 30 19:07:07.283611 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:07.283584 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 19:07:07.283670 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:07.283647 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 19:07:07.283720 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:07.283677 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 19:07:07.352035 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:07.352013 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-wfw84" Oct 30 19:07:07.644806 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:07.644781 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-wfw84"] Oct 30 19:07:07.645479 ip-10-0-140-163 kubenswrapper[6960]: W1030 19:07:07.645454 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda02a9d7_7b75_401c_9134_8e45d7303175.slice/crio-97cd57819c4c15659434b0a7d7ba5b48343daca34cce1c74ada39251918e515e WatchSource:0}: Error finding container 97cd57819c4c15659434b0a7d7ba5b48343daca34cce1c74ada39251918e515e: Status 404 returned error can't find the container with id 97cd57819c4c15659434b0a7d7ba5b48343daca34cce1c74ada39251918e515e Oct 30 19:07:07.648525 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:07.648496 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 19:07:07.648617 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:07.648559 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 19:07:07.648617 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:07.648589 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 19:07:07.664776 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:07.664750 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-qhvj4" Oct 30 19:07:07.665349 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:07.665317 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-qhvj4" Oct 30 19:07:07.667254 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:07.666592 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-qhvj4" Oct 30 19:07:07.746086 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:07.746047 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-689c23-58c9d7c564-ncqml"] Oct 30 19:07:07.752431 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:07.752409 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-689c23-58c9d7c564-ncqml" Oct 30 19:07:07.760266 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:07.760010 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-689c23-serving-cert\"" Oct 30 19:07:07.787063 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:07.787023 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-689c23-58c9d7c564-ncqml"] Oct 30 19:07:07.806306 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:07.806109 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:07.827555 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:07.827232 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ac809bd-4083-4519-b7b5-903a22885985-maas-api-tls\") pod \"maas-api-e2e-extmod-a-689c23-58c9d7c564-ncqml\" (UID: \"0ac809bd-4083-4519-b7b5-903a22885985\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-689c23-58c9d7c564-ncqml" Oct 30 19:07:07.827886 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:07.827851 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmrt4\" (UniqueName: \"kubernetes.io/projected/0ac809bd-4083-4519-b7b5-903a22885985-kube-api-access-kmrt4\") pod \"maas-api-e2e-extmod-a-689c23-58c9d7c564-ncqml\" (UID: \"0ac809bd-4083-4519-b7b5-903a22885985\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-689c23-58c9d7c564-ncqml" Oct 30 19:07:07.828046 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:07.828027 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ac809bd-4083-4519-b7b5-903a22885985-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-689c23-58c9d7c564-ncqml\" (UID: \"0ac809bd-4083-4519-b7b5-903a22885985\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-689c23-58c9d7c564-ncqml" Oct 30 19:07:07.828881 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:07.828165 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ac809bd-4083-4519-b7b5-903a22885985-metrics-certs\") pod \"maas-api-e2e-extmod-a-689c23-58c9d7c564-ncqml\" (UID: \"0ac809bd-4083-4519-b7b5-903a22885985\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-689c23-58c9d7c564-ncqml" Oct 30 19:07:07.904973 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:07.904888 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-wfw84" event={"ID":"da02a9d7-7b75-401c-9134-8e45d7303175","Type":"ContainerStarted","Data":"30633a687fadded7a1baf9dd9d6a60ebd1a07128924ad553e4a1430a0ccfc5c1"} Oct 30 19:07:07.904973 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:07.904958 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-wfw84" event={"ID":"da02a9d7-7b75-401c-9134-8e45d7303175","Type":"ContainerStarted","Data":"97cd57819c4c15659434b0a7d7ba5b48343daca34cce1c74ada39251918e515e"} Oct 30 19:07:07.909678 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:07.909649 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-pzmg6" event={"ID":"50c12729-1387-436a-bee0-9eb3316db708","Type":"ContainerStarted","Data":"25e863cbfc452efebbfa326f7074d5a7e3e3bcb51c9a80090e8d261dc45d26d3"} Oct 30 19:07:07.909915 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:07.909894 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-pzmg6" event={"ID":"50c12729-1387-436a-bee0-9eb3316db708","Type":"ContainerStarted","Data":"b8887ac9dbd301fd7badfb0721c9ed2d6dc611b2b7ccf7d05aab7c4b823f5146"} Oct 30 19:07:07.923213 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:07.921093 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:07.931046 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:07.930957 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmrt4\" (UniqueName: \"kubernetes.io/projected/0ac809bd-4083-4519-b7b5-903a22885985-kube-api-access-kmrt4\") pod \"maas-api-e2e-extmod-a-689c23-58c9d7c564-ncqml\" (UID: \"0ac809bd-4083-4519-b7b5-903a22885985\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-689c23-58c9d7c564-ncqml" Oct 30 19:07:07.931492 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:07.931456 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ac809bd-4083-4519-b7b5-903a22885985-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-689c23-58c9d7c564-ncqml\" (UID: \"0ac809bd-4083-4519-b7b5-903a22885985\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-689c23-58c9d7c564-ncqml" Oct 30 19:07:07.931770 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:07.931753 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ac809bd-4083-4519-b7b5-903a22885985-metrics-certs\") pod \"maas-api-e2e-extmod-a-689c23-58c9d7c564-ncqml\" (UID: \"0ac809bd-4083-4519-b7b5-903a22885985\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-689c23-58c9d7c564-ncqml" Oct 30 19:07:07.933216 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:07.932256 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ac809bd-4083-4519-b7b5-903a22885985-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-689c23-58c9d7c564-ncqml\" (UID: \"0ac809bd-4083-4519-b7b5-903a22885985\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-689c23-58c9d7c564-ncqml" Oct 30 19:07:07.933216 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:07.932719 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ac809bd-4083-4519-b7b5-903a22885985-maas-api-tls\") pod \"maas-api-e2e-extmod-a-689c23-58c9d7c564-ncqml\" (UID: \"0ac809bd-4083-4519-b7b5-903a22885985\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-689c23-58c9d7c564-ncqml" Oct 30 19:07:07.941473 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:07.940936 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-wfw84" podStartSLOduration=1.940919786 podStartE2EDuration="1.940919786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:07:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:07:07.937367555 +0000 UTC m=+1995.466398694" watchObservedRunningTime="2026-10-30 19:07:07.940919786 +0000 UTC m=+1995.469950925" Oct 30 19:07:07.943646 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:07.943619 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-pzmg6" Oct 30 19:07:07.944594 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:07.944571 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ac809bd-4083-4519-b7b5-903a22885985-maas-api-tls\") pod \"maas-api-e2e-extmod-a-689c23-58c9d7c564-ncqml\" (UID: \"0ac809bd-4083-4519-b7b5-903a22885985\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-689c23-58c9d7c564-ncqml" Oct 30 19:07:07.945138 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:07.945118 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ac809bd-4083-4519-b7b5-903a22885985-metrics-certs\") pod \"maas-api-e2e-extmod-a-689c23-58c9d7c564-ncqml\" (UID: \"0ac809bd-4083-4519-b7b5-903a22885985\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-689c23-58c9d7c564-ncqml" Oct 30 19:07:07.949678 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:07.949656 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmrt4\" (UniqueName: \"kubernetes.io/projected/0ac809bd-4083-4519-b7b5-903a22885985-kube-api-access-kmrt4\") pod \"maas-api-e2e-extmod-a-689c23-58c9d7c564-ncqml\" (UID: \"0ac809bd-4083-4519-b7b5-903a22885985\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-689c23-58c9d7c564-ncqml" Oct 30 19:07:07.967230 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:07.965189 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-pzmg6" podStartSLOduration=1.965171474 podStartE2EDuration="1.965171474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:07:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:07:07.961671272 +0000 UTC m=+1995.490702412" watchObservedRunningTime="2026-10-30 19:07:07.965171474 +0000 UTC m=+1995.494202614" Oct 30 19:07:08.042590 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:08.042556 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:08.107087 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:08.107059 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-689c23-58c9d7c564-ncqml" Oct 30 19:07:08.357708 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:08.357674 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-wfw84" Oct 30 19:07:08.368328 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:08.368292 6960 patch_prober.go:29] interesting pod/e2e-extmod-b-d72beb-openshift-default-869db9998f-wfw84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 30 19:07:08.368648 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:08.368614 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-wfw84" podUID="da02a9d7-7b75-401c-9134-8e45d7303175" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 30 19:07:08.440778 ip-10-0-140-163 kubenswrapper[6960]: W1030 19:07:08.440746 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ac809bd_4083_4519_b7b5_903a22885985.slice/crio-a18bee4d014c8dc17cc21ef0809c5c912d4331855b90b56bad924a340d7467ae WatchSource:0}: Error finding container a18bee4d014c8dc17cc21ef0809c5c912d4331855b90b56bad924a340d7467ae: Status 404 returned error can't find the container with id a18bee4d014c8dc17cc21ef0809c5c912d4331855b90b56bad924a340d7467ae Oct 30 19:07:08.443248 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:08.443220 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-689c23-58c9d7c564-ncqml"] Oct 30 19:07:08.923879 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:08.923840 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-689c23-58c9d7c564-ncqml" event={"ID":"0ac809bd-4083-4519-b7b5-903a22885985","Type":"ContainerStarted","Data":"a18bee4d014c8dc17cc21ef0809c5c912d4331855b90b56bad924a340d7467ae"} Oct 30 19:07:08.944645 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:08.944608 6960 patch_prober.go:29] interesting pod/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-pzmg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 30 19:07:08.944741 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:08.944658 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-pzmg6" podUID="50c12729-1387-436a-bee0-9eb3316db708" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 30 19:07:09.014419 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:09.014385 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:09.352589 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:09.352555 6960 patch_prober.go:29] interesting pod/e2e-extmod-b-d72beb-openshift-default-869db9998f-wfw84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 30 19:07:09.352767 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:09.352616 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-wfw84" podUID="da02a9d7-7b75-401c-9134-8e45d7303175" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 30 19:07:09.933936 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:09.933901 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-689c23-58c9d7c564-ncqml" event={"ID":"0ac809bd-4083-4519-b7b5-903a22885985","Type":"ContainerStarted","Data":"6e9eb76380dd6c618dff8789cc7a5974398567bb6326177c3fe33528745450d5"} Oct 30 19:07:09.934355 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:09.934004 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-689c23-58c9d7c564-ncqml" Oct 30 19:07:09.943058 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:09.943037 6960 patch_prober.go:29] interesting pod/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-pzmg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 30 19:07:09.943164 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:09.943075 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-pzmg6" podUID="50c12729-1387-436a-bee0-9eb3316db708" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 30 19:07:09.983171 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:09.982611 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-689c23-58c9d7c564-ncqml" podStartSLOduration=2.487397696 podStartE2EDuration="2.98259185s" totalImagesPullingTime="495.194154ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:07:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:07:09.963564174 +0000 UTC m=+1997.492595313" watchObservedRunningTime="2026-10-30 19:07:09.98259185 +0000 UTC m=+1997.511622989" Oct 30 19:07:10.241957 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:10.241868 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:10.353318 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:10.353283 6960 patch_prober.go:29] interesting pod/e2e-extmod-b-d72beb-openshift-default-869db9998f-wfw84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 30 19:07:10.353534 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:10.353339 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-wfw84" podUID="da02a9d7-7b75-401c-9134-8e45d7303175" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 30 19:07:10.942674 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:10.942643 6960 patch_prober.go:29] interesting pod/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-pzmg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 30 19:07:10.943076 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:10.942690 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-pzmg6" podUID="50c12729-1387-436a-bee0-9eb3316db708" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 30 19:07:11.209027 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:11.208537 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:11.352424 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:11.352390 6960 patch_prober.go:29] interesting pod/e2e-extmod-b-d72beb-openshift-default-869db9998f-wfw84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 30 19:07:11.352600 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:11.352450 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-wfw84" podUID="da02a9d7-7b75-401c-9134-8e45d7303175" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 30 19:07:11.943136 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:11.943102 6960 patch_prober.go:29] interesting pod/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-pzmg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 30 19:07:11.943637 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:11.943192 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-pzmg6" podUID="50c12729-1387-436a-bee0-9eb3316db708" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 30 19:07:12.320821 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:12.320782 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:12.352863 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:12.352714 6960 patch_prober.go:29] interesting pod/e2e-extmod-b-d72beb-openshift-default-869db9998f-wfw84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 30 19:07:12.352863 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:12.352777 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-wfw84" podUID="da02a9d7-7b75-401c-9134-8e45d7303175" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 30 19:07:12.943027 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:12.942992 6960 patch_prober.go:29] interesting pod/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-pzmg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 30 19:07:12.943285 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:12.943047 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-pzmg6" podUID="50c12729-1387-436a-bee0-9eb3316db708" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 30 19:07:13.353369 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:13.353340 6960 patch_prober.go:29] interesting pod/e2e-extmod-b-d72beb-openshift-default-869db9998f-wfw84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 30 19:07:13.353555 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:13.353394 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-wfw84" podUID="da02a9d7-7b75-401c-9134-8e45d7303175" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 30 19:07:13.425503 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:13.425470 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:13.942712 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:13.942673 6960 patch_prober.go:29] interesting pod/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-pzmg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 30 19:07:13.942937 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:13.942734 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-pzmg6" podUID="50c12729-1387-436a-bee0-9eb3316db708" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 30 19:07:14.323529 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:14.323486 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:14.352854 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:14.352809 6960 patch_prober.go:29] interesting pod/e2e-extmod-b-d72beb-openshift-default-869db9998f-wfw84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 30 19:07:14.353062 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:14.352877 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-wfw84" podUID="da02a9d7-7b75-401c-9134-8e45d7303175" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 30 19:07:14.948486 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:14.948455 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-pzmg6" Oct 30 19:07:14.948873 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:14.948856 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-pzmg6" Oct 30 19:07:14.949567 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:14.949548 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-pzmg6" Oct 30 19:07:15.359055 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:15.359025 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-wfw84" Oct 30 19:07:15.359535 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:15.359515 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-wfw84" Oct 30 19:07:15.360270 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:15.360253 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-wfw84" Oct 30 19:07:15.370802 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:15.370770 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:15.947726 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:15.947694 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-689c23-58c9d7c564-ncqml" Oct 30 19:07:16.360682 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:16.360637 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:17.128747 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:17.128713 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:18.010908 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:18.010869 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:18.796877 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:18.796838 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:19.678439 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:19.678408 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:20.431644 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:20.431611 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:21.202274 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:21.202233 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:21.986871 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:21.986841 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-mw9h7"] Oct 30 19:07:21.991571 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:21.991552 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-mw9h7" Oct 30 19:07:22.005876 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:22.005846 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-mw9h7"] Oct 30 19:07:22.030043 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:22.030014 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:22.036871 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:22.036840 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afe56fb6-3076-47f1-92c7-a45b0426bdad-credential-socket\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-mw9h7\" (UID: \"afe56fb6-3076-47f1-92c7-a45b0426bdad\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-mw9h7" Oct 30 19:07:22.037015 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:22.036895 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afe56fb6-3076-47f1-92c7-a45b0426bdad-istio-envoy\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-mw9h7\" (UID: \"afe56fb6-3076-47f1-92c7-a45b0426bdad\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-mw9h7" Oct 30 19:07:22.037015 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:22.037001 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afe56fb6-3076-47f1-92c7-a45b0426bdad-workload-certs\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-mw9h7\" (UID: \"afe56fb6-3076-47f1-92c7-a45b0426bdad\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-mw9h7" Oct 30 19:07:22.037165 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:22.037039 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7htz9\" (UniqueName: \"kubernetes.io/projected/afe56fb6-3076-47f1-92c7-a45b0426bdad-kube-api-access-7htz9\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-mw9h7\" (UID: \"afe56fb6-3076-47f1-92c7-a45b0426bdad\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-mw9h7" Oct 30 19:07:22.037233 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:22.037212 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afe56fb6-3076-47f1-92c7-a45b0426bdad-istio-token\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-mw9h7\" (UID: \"afe56fb6-3076-47f1-92c7-a45b0426bdad\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-mw9h7" Oct 30 19:07:22.037293 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:22.037246 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afe56fb6-3076-47f1-92c7-a45b0426bdad-istio-podinfo\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-mw9h7\" (UID: \"afe56fb6-3076-47f1-92c7-a45b0426bdad\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-mw9h7" Oct 30 19:07:22.037293 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:22.037285 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afe56fb6-3076-47f1-92c7-a45b0426bdad-workload-socket\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-mw9h7\" (UID: \"afe56fb6-3076-47f1-92c7-a45b0426bdad\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-mw9h7" Oct 30 19:07:22.037479 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:22.037436 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afe56fb6-3076-47f1-92c7-a45b0426bdad-istiod-ca-cert\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-mw9h7\" (UID: \"afe56fb6-3076-47f1-92c7-a45b0426bdad\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-mw9h7" Oct 30 19:07:22.037708 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:22.037675 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afe56fb6-3076-47f1-92c7-a45b0426bdad-istio-data\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-mw9h7\" (UID: \"afe56fb6-3076-47f1-92c7-a45b0426bdad\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-mw9h7" Oct 30 19:07:22.138778 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:22.138745 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afe56fb6-3076-47f1-92c7-a45b0426bdad-istio-token\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-mw9h7\" (UID: \"afe56fb6-3076-47f1-92c7-a45b0426bdad\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-mw9h7" Oct 30 19:07:22.138778 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:22.138786 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afe56fb6-3076-47f1-92c7-a45b0426bdad-istio-podinfo\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-mw9h7\" (UID: \"afe56fb6-3076-47f1-92c7-a45b0426bdad\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-mw9h7" Oct 30 19:07:22.139017 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:22.138830 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afe56fb6-3076-47f1-92c7-a45b0426bdad-workload-socket\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-mw9h7\" (UID: \"afe56fb6-3076-47f1-92c7-a45b0426bdad\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-mw9h7" Oct 30 19:07:22.139017 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:22.138870 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afe56fb6-3076-47f1-92c7-a45b0426bdad-istiod-ca-cert\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-mw9h7\" (UID: \"afe56fb6-3076-47f1-92c7-a45b0426bdad\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-mw9h7" Oct 30 19:07:22.139017 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:22.138927 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afe56fb6-3076-47f1-92c7-a45b0426bdad-istio-data\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-mw9h7\" (UID: \"afe56fb6-3076-47f1-92c7-a45b0426bdad\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-mw9h7" Oct 30 19:07:22.139198 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:22.139083 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afe56fb6-3076-47f1-92c7-a45b0426bdad-credential-socket\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-mw9h7\" (UID: \"afe56fb6-3076-47f1-92c7-a45b0426bdad\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-mw9h7" Oct 30 19:07:22.139198 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:22.139122 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afe56fb6-3076-47f1-92c7-a45b0426bdad-istio-envoy\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-mw9h7\" (UID: \"afe56fb6-3076-47f1-92c7-a45b0426bdad\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-mw9h7" Oct 30 19:07:22.139322 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:22.139197 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afe56fb6-3076-47f1-92c7-a45b0426bdad-workload-certs\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-mw9h7\" (UID: \"afe56fb6-3076-47f1-92c7-a45b0426bdad\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-mw9h7" Oct 30 19:07:22.139322 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:22.139235 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7htz9\" (UniqueName: \"kubernetes.io/projected/afe56fb6-3076-47f1-92c7-a45b0426bdad-kube-api-access-7htz9\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-mw9h7\" (UID: \"afe56fb6-3076-47f1-92c7-a45b0426bdad\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-mw9h7" Oct 30 19:07:22.139440 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:22.139376 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afe56fb6-3076-47f1-92c7-a45b0426bdad-workload-socket\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-mw9h7\" (UID: \"afe56fb6-3076-47f1-92c7-a45b0426bdad\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-mw9h7" Oct 30 19:07:22.139522 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:22.139492 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afe56fb6-3076-47f1-92c7-a45b0426bdad-credential-socket\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-mw9h7\" (UID: \"afe56fb6-3076-47f1-92c7-a45b0426bdad\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-mw9h7" Oct 30 19:07:22.139782 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:22.139760 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afe56fb6-3076-47f1-92c7-a45b0426bdad-istiod-ca-cert\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-mw9h7\" (UID: \"afe56fb6-3076-47f1-92c7-a45b0426bdad\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-mw9h7" Oct 30 19:07:22.139875 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:22.139773 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afe56fb6-3076-47f1-92c7-a45b0426bdad-workload-certs\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-mw9h7\" (UID: \"afe56fb6-3076-47f1-92c7-a45b0426bdad\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-mw9h7" Oct 30 19:07:22.140332 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:22.140312 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afe56fb6-3076-47f1-92c7-a45b0426bdad-istio-data\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-mw9h7\" (UID: \"afe56fb6-3076-47f1-92c7-a45b0426bdad\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-mw9h7" Oct 30 19:07:22.141969 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:22.141948 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afe56fb6-3076-47f1-92c7-a45b0426bdad-istio-podinfo\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-mw9h7\" (UID: \"afe56fb6-3076-47f1-92c7-a45b0426bdad\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-mw9h7" Oct 30 19:07:22.142260 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:22.142237 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afe56fb6-3076-47f1-92c7-a45b0426bdad-istio-envoy\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-mw9h7\" (UID: \"afe56fb6-3076-47f1-92c7-a45b0426bdad\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-mw9h7" Oct 30 19:07:22.152665 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:22.152645 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afe56fb6-3076-47f1-92c7-a45b0426bdad-istio-token\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-mw9h7\" (UID: \"afe56fb6-3076-47f1-92c7-a45b0426bdad\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-mw9h7" Oct 30 19:07:22.156518 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:22.156488 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7htz9\" (UniqueName: \"kubernetes.io/projected/afe56fb6-3076-47f1-92c7-a45b0426bdad-kube-api-access-7htz9\") pod \"e2e-extmod-b-d72beb-openshift-default-869db9998f-mw9h7\" (UID: \"afe56fb6-3076-47f1-92c7-a45b0426bdad\") " pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-mw9h7" Oct 30 19:07:22.308024 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:22.307994 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-mw9h7" Oct 30 19:07:22.596537 ip-10-0-140-163 kubenswrapper[6960]: W1030 19:07:22.596504 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafe56fb6_3076_47f1_92c7_a45b0426bdad.slice/crio-d9210f751ae600e9936eb62d0bfff862c735ee1c0aba660557636f5dc9f902a2 WatchSource:0}: Error finding container d9210f751ae600e9936eb62d0bfff862c735ee1c0aba660557636f5dc9f902a2: Status 404 returned error can't find the container with id d9210f751ae600e9936eb62d0bfff862c735ee1c0aba660557636f5dc9f902a2 Oct 30 19:07:22.597841 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:22.597816 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-mw9h7"] Oct 30 19:07:22.599648 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:22.599611 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 19:07:22.599757 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:22.599688 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 19:07:22.599757 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:22.599729 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 19:07:22.841957 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:22.841908 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:23.065192 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:23.065118 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-mw9h7" event={"ID":"afe56fb6-3076-47f1-92c7-a45b0426bdad","Type":"ContainerStarted","Data":"bfc0d3e2d12d5641b944a03b84362686bb3b0a7c50109ae5a6b82bf1ec158bc6"} Oct 30 19:07:23.065423 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:23.065226 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-mw9h7" event={"ID":"afe56fb6-3076-47f1-92c7-a45b0426bdad","Type":"ContainerStarted","Data":"d9210f751ae600e9936eb62d0bfff862c735ee1c0aba660557636f5dc9f902a2"} Oct 30 19:07:23.093819 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:23.093763 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-mw9h7" podStartSLOduration=2.093744911 podStartE2EDuration="2.093744911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:07:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:07:23.087968484 +0000 UTC m=+2010.616999622" watchObservedRunningTime="2026-10-30 19:07:23.093744911 +0000 UTC m=+2010.622776051" Oct 30 19:07:23.308628 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:23.308595 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-mw9h7" Oct 30 19:07:23.309809 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:23.309785 6960 patch_prober.go:29] interesting pod/e2e-extmod-b-d72beb-openshift-default-869db9998f-mw9h7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 30 19:07:23.309931 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:23.309843 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-mw9h7" podUID="afe56fb6-3076-47f1-92c7-a45b0426bdad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 30 19:07:23.723202 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:23.723136 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:23.934031 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:07:23.933988 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 19:07:24.309224 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:24.309190 6960 patch_prober.go:29] interesting pod/e2e-extmod-b-d72beb-openshift-default-869db9998f-mw9h7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 30 19:07:24.309608 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:24.309250 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-mw9h7" podUID="afe56fb6-3076-47f1-92c7-a45b0426bdad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 30 19:07:24.632554 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:24.632468 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:25.309011 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:25.308971 6960 patch_prober.go:29] interesting pod/e2e-extmod-b-d72beb-openshift-default-869db9998f-mw9h7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 30 19:07:25.309230 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:25.309037 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-mw9h7" podUID="afe56fb6-3076-47f1-92c7-a45b0426bdad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 30 19:07:25.606258 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:25.606171 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:26.308927 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:26.308885 6960 patch_prober.go:29] interesting pod/e2e-extmod-b-d72beb-openshift-default-869db9998f-mw9h7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 30 19:07:26.309163 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:26.308948 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-mw9h7" podUID="afe56fb6-3076-47f1-92c7-a45b0426bdad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 30 19:07:26.580954 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:26.578714 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:27.308537 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:27.308503 6960 patch_prober.go:29] interesting pod/e2e-extmod-b-d72beb-openshift-default-869db9998f-mw9h7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 30 19:07:27.308929 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:27.308559 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-mw9h7" podUID="afe56fb6-3076-47f1-92c7-a45b0426bdad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 30 19:07:27.442610 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:27.442574 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:28.287409 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:28.287376 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:28.308879 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:28.308849 6960 patch_prober.go:29] interesting pod/e2e-extmod-b-d72beb-openshift-default-869db9998f-mw9h7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 30 19:07:28.309276 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:28.308908 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-mw9h7" podUID="afe56fb6-3076-47f1-92c7-a45b0426bdad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 30 19:07:29.144178 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:29.144077 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:29.309037 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:29.309005 6960 patch_prober.go:29] interesting pod/e2e-extmod-b-d72beb-openshift-default-869db9998f-mw9h7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 30 19:07:29.309471 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:29.309057 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-mw9h7" podUID="afe56fb6-3076-47f1-92c7-a45b0426bdad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 30 19:07:30.121518 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:30.121477 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:30.313243 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:30.313214 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-mw9h7" Oct 30 19:07:30.313745 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:30.313727 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-mw9h7" Oct 30 19:07:30.314587 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:30.314569 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-mw9h7" Oct 30 19:07:30.984978 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:30.984940 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:31.925269 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:31.925234 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:33.212848 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:33.212815 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:34.206134 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:34.206103 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:35.175597 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:35.175558 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:35.298005 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:35.297969 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:36.314782 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:36.314744 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:36.765854 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:36.765764 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:37.321493 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:37.321450 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:37.503157 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:37.503107 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:38.020777 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:38.020744 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:38.596086 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:38.596050 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:39.135934 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:39.135894 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:39.297668 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:39.297633 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:39.463095 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:39.463007 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:39.704262 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:39.704230 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:40.001008 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:40.000974 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:40.818053 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:40.818015 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:41.000479 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:41.000442 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:41.298224 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:41.291793 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:41.406967 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:41.406928 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:41.796882 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:41.796840 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:42.049760 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:42.049669 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:42.212166 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:42.212100 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:43.079417 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:43.079372 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:43.390514 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:43.390422 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:43.537797 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:43.537766 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:44.407177 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:44.405900 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:45.268169 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:45.260930 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:46.219368 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:46.219333 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:47.122443 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:47.122402 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:48.032406 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:48.032370 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:48.931720 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:48.931677 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:50.011707 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:50.011667 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:50.825198 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:50.825130 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:51.825465 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:51.825426 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:52.706613 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:52.706575 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:53.327652 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:53.327614 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:53.522106 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:53.522074 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:53.936752 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:07:53.936715 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 19:07:54.228755 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:54.228671 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-5f7lv"] Oct 30 19:07:54.234794 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:54.234752 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-5f7lv" Oct 30 19:07:54.263013 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:54.262979 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-5f7lv"] Oct 30 19:07:54.325529 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:54.325497 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-689c23-openshift-default-dockercfg-ng4kn\"" Oct 30 19:07:54.368510 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:54.368469 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/755750a6-9b02-43c7-adfd-00f2073020fe-istio-token\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-5f7lv\" (UID: \"755750a6-9b02-43c7-adfd-00f2073020fe\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-5f7lv" Oct 30 19:07:54.368952 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:54.368698 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/755750a6-9b02-43c7-adfd-00f2073020fe-credential-socket\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-5f7lv\" (UID: \"755750a6-9b02-43c7-adfd-00f2073020fe\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-5f7lv" Oct 30 19:07:54.368952 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:54.368767 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdgvq\" (UniqueName: \"kubernetes.io/projected/755750a6-9b02-43c7-adfd-00f2073020fe-kube-api-access-gdgvq\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-5f7lv\" (UID: \"755750a6-9b02-43c7-adfd-00f2073020fe\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-5f7lv" Oct 30 19:07:54.368952 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:54.368832 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/755750a6-9b02-43c7-adfd-00f2073020fe-istio-podinfo\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-5f7lv\" (UID: \"755750a6-9b02-43c7-adfd-00f2073020fe\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-5f7lv" Oct 30 19:07:54.368952 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:54.368920 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/755750a6-9b02-43c7-adfd-00f2073020fe-workload-certs\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-5f7lv\" (UID: \"755750a6-9b02-43c7-adfd-00f2073020fe\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-5f7lv" Oct 30 19:07:54.368952 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:54.368953 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/755750a6-9b02-43c7-adfd-00f2073020fe-istio-envoy\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-5f7lv\" (UID: \"755750a6-9b02-43c7-adfd-00f2073020fe\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-5f7lv" Oct 30 19:07:54.369219 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:54.369007 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/755750a6-9b02-43c7-adfd-00f2073020fe-workload-socket\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-5f7lv\" (UID: \"755750a6-9b02-43c7-adfd-00f2073020fe\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-5f7lv" Oct 30 19:07:54.369219 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:54.369039 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/755750a6-9b02-43c7-adfd-00f2073020fe-istiod-ca-cert\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-5f7lv\" (UID: \"755750a6-9b02-43c7-adfd-00f2073020fe\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-5f7lv" Oct 30 19:07:54.369219 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:54.369125 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/755750a6-9b02-43c7-adfd-00f2073020fe-istio-data\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-5f7lv\" (UID: \"755750a6-9b02-43c7-adfd-00f2073020fe\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-5f7lv" Oct 30 19:07:54.436117 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:54.436081 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:54.471118 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:54.471070 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/755750a6-9b02-43c7-adfd-00f2073020fe-credential-socket\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-5f7lv\" (UID: \"755750a6-9b02-43c7-adfd-00f2073020fe\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-5f7lv" Oct 30 19:07:54.471386 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:54.471231 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdgvq\" (UniqueName: \"kubernetes.io/projected/755750a6-9b02-43c7-adfd-00f2073020fe-kube-api-access-gdgvq\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-5f7lv\" (UID: \"755750a6-9b02-43c7-adfd-00f2073020fe\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-5f7lv" Oct 30 19:07:54.471386 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:54.471266 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/755750a6-9b02-43c7-adfd-00f2073020fe-istio-podinfo\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-5f7lv\" (UID: \"755750a6-9b02-43c7-adfd-00f2073020fe\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-5f7lv" Oct 30 19:07:54.471386 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:54.471297 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/755750a6-9b02-43c7-adfd-00f2073020fe-workload-certs\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-5f7lv\" (UID: \"755750a6-9b02-43c7-adfd-00f2073020fe\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-5f7lv" Oct 30 19:07:54.471637 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:54.471614 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/755750a6-9b02-43c7-adfd-00f2073020fe-istio-envoy\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-5f7lv\" (UID: \"755750a6-9b02-43c7-adfd-00f2073020fe\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-5f7lv" Oct 30 19:07:54.471720 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:54.471660 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/755750a6-9b02-43c7-adfd-00f2073020fe-workload-socket\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-5f7lv\" (UID: \"755750a6-9b02-43c7-adfd-00f2073020fe\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-5f7lv" Oct 30 19:07:54.471780 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:54.471686 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/755750a6-9b02-43c7-adfd-00f2073020fe-istiod-ca-cert\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-5f7lv\" (UID: \"755750a6-9b02-43c7-adfd-00f2073020fe\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-5f7lv" Oct 30 19:07:54.471838 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:54.471812 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/755750a6-9b02-43c7-adfd-00f2073020fe-istio-data\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-5f7lv\" (UID: \"755750a6-9b02-43c7-adfd-00f2073020fe\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-5f7lv" Oct 30 19:07:54.489844 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:54.471930 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/755750a6-9b02-43c7-adfd-00f2073020fe-istio-token\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-5f7lv\" (UID: \"755750a6-9b02-43c7-adfd-00f2073020fe\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-5f7lv" Oct 30 19:07:54.489844 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:54.472850 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/755750a6-9b02-43c7-adfd-00f2073020fe-istiod-ca-cert\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-5f7lv\" (UID: \"755750a6-9b02-43c7-adfd-00f2073020fe\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-5f7lv" Oct 30 19:07:54.489844 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:54.473253 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/755750a6-9b02-43c7-adfd-00f2073020fe-istio-data\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-5f7lv\" (UID: \"755750a6-9b02-43c7-adfd-00f2073020fe\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-5f7lv" Oct 30 19:07:54.489844 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:54.473507 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/755750a6-9b02-43c7-adfd-00f2073020fe-credential-socket\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-5f7lv\" (UID: \"755750a6-9b02-43c7-adfd-00f2073020fe\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-5f7lv" Oct 30 19:07:54.489844 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:54.473802 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/755750a6-9b02-43c7-adfd-00f2073020fe-workload-certs\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-5f7lv\" (UID: \"755750a6-9b02-43c7-adfd-00f2073020fe\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-5f7lv" Oct 30 19:07:54.489844 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:54.474461 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/755750a6-9b02-43c7-adfd-00f2073020fe-workload-socket\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-5f7lv\" (UID: \"755750a6-9b02-43c7-adfd-00f2073020fe\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-5f7lv" Oct 30 19:07:54.495469 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:54.495436 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/755750a6-9b02-43c7-adfd-00f2073020fe-istio-envoy\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-5f7lv\" (UID: \"755750a6-9b02-43c7-adfd-00f2073020fe\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-5f7lv" Oct 30 19:07:54.496470 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:54.496442 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdgvq\" (UniqueName: \"kubernetes.io/projected/755750a6-9b02-43c7-adfd-00f2073020fe-kube-api-access-gdgvq\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-5f7lv\" (UID: \"755750a6-9b02-43c7-adfd-00f2073020fe\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-5f7lv" Oct 30 19:07:54.501706 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:54.501679 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/755750a6-9b02-43c7-adfd-00f2073020fe-istio-podinfo\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-5f7lv\" (UID: \"755750a6-9b02-43c7-adfd-00f2073020fe\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-5f7lv" Oct 30 19:07:54.510865 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:54.510833 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/755750a6-9b02-43c7-adfd-00f2073020fe-istio-token\") pod \"e2e-extmod-a-689c23-openshift-default-57dff78645-5f7lv\" (UID: \"755750a6-9b02-43c7-adfd-00f2073020fe\") " pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-5f7lv" Oct 30 19:07:54.565249 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:54.565213 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-5f7lv" Oct 30 19:07:54.873612 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:54.873586 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-5f7lv"] Oct 30 19:07:54.874173 ip-10-0-140-163 kubenswrapper[6960]: W1030 19:07:54.874134 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod755750a6_9b02_43c7_adfd_00f2073020fe.slice/crio-1feec519a939586556f8852217cc758fe9317bebbe6247d7cf21cdac9c408382 WatchSource:0}: Error finding container 1feec519a939586556f8852217cc758fe9317bebbe6247d7cf21cdac9c408382: Status 404 returned error can't find the container with id 1feec519a939586556f8852217cc758fe9317bebbe6247d7cf21cdac9c408382 Oct 30 19:07:54.877073 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:54.877039 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 19:07:54.877201 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:54.877105 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 19:07:54.877201 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:54.877133 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 19:07:55.316957 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:55.316925 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:55.349206 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:55.349170 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-5f7lv" event={"ID":"755750a6-9b02-43c7-adfd-00f2073020fe","Type":"ContainerStarted","Data":"f6a1724d2ff162b697d3d8fd0bd2737e9383a6b193a237e7539db8eb400a6902"} Oct 30 19:07:55.349377 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:55.349234 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-5f7lv" event={"ID":"755750a6-9b02-43c7-adfd-00f2073020fe","Type":"ContainerStarted","Data":"1feec519a939586556f8852217cc758fe9317bebbe6247d7cf21cdac9c408382"} Oct 30 19:07:55.376896 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:55.376818 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-5f7lv" podStartSLOduration=1.376797904 podStartE2EDuration="1.376797904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:07:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:07:55.373179574 +0000 UTC m=+2042.902210716" watchObservedRunningTime="2026-10-30 19:07:55.376797904 +0000 UTC m=+2042.905829043" Oct 30 19:07:55.568066 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:55.567981 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-5f7lv" Oct 30 19:07:55.569718 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:55.569694 6960 patch_prober.go:29] interesting pod/e2e-extmod-a-689c23-openshift-default-57dff78645-5f7lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 30 19:07:55.569860 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:55.569733 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-5f7lv" podUID="755750a6-9b02-43c7-adfd-00f2073020fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 30 19:07:56.436517 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:56.436480 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:56.607175 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:56.600241 6960 patch_prober.go:29] interesting pod/e2e-extmod-a-689c23-openshift-default-57dff78645-5f7lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 30 19:07:56.607175 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:56.600305 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-5f7lv" podUID="755750a6-9b02-43c7-adfd-00f2073020fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 30 19:07:57.420354 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:57.420307 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:57.579612 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:57.579534 6960 patch_prober.go:29] interesting pod/e2e-extmod-a-689c23-openshift-default-57dff78645-5f7lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 30 19:07:57.579612 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:57.579592 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-5f7lv" podUID="755750a6-9b02-43c7-adfd-00f2073020fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 30 19:07:58.512696 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:58.512641 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:07:58.573202 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:58.569712 6960 patch_prober.go:29] interesting pod/e2e-extmod-a-689c23-openshift-default-57dff78645-5f7lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 30 19:07:58.573202 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:58.569776 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-5f7lv" podUID="755750a6-9b02-43c7-adfd-00f2073020fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 30 19:07:59.570009 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:59.569507 6960 patch_prober.go:29] interesting pod/e2e-extmod-a-689c23-openshift-default-57dff78645-5f7lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 30 19:07:59.570009 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:07:59.569569 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-5f7lv" podUID="755750a6-9b02-43c7-adfd-00f2073020fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 30 19:08:00.081057 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:00.081025 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:00.576281 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:00.568501 6960 patch_prober.go:29] interesting pod/e2e-extmod-a-689c23-openshift-default-57dff78645-5f7lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 30 19:08:00.576281 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:00.568563 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-5f7lv" podUID="755750a6-9b02-43c7-adfd-00f2073020fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 30 19:08:00.988212 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:00.985484 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:01.665194 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:01.665125 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-5f7lv" Oct 30 19:08:01.666005 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:01.665982 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-5f7lv" Oct 30 19:08:01.666772 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:01.666749 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-5f7lv" Oct 30 19:08:01.894379 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:01.894346 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:02.822989 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:02.822944 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:03.635107 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:03.635068 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:03.858344 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:03.858305 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:03.973839 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:03.973733 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:05.181679 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:05.181644 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:05.982746 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:05.982711 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:06.763581 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:06.763549 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-e77b1d-6c5d97c974-5r5f8"] Oct 30 19:08:06.769486 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:06.769464 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e77b1d-6c5d97c974-5r5f8" Oct 30 19:08:06.776949 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:06.776902 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-e77b1d-serving-cert\"" Oct 30 19:08:06.780639 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:06.780611 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-e77b1d-6c5d97c974-5r5f8"] Oct 30 19:08:06.816073 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:06.816036 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56a9fce0-2511-4217-afb8-feb2184fb974-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-e77b1d-6c5d97c974-5r5f8\" (UID: \"56a9fce0-2511-4217-afb8-feb2184fb974\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e77b1d-6c5d97c974-5r5f8" Oct 30 19:08:06.816364 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:06.816342 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56a9fce0-2511-4217-afb8-feb2184fb974-metrics-certs\") pod \"maas-api-e2e-extmod-b-e77b1d-6c5d97c974-5r5f8\" (UID: \"56a9fce0-2511-4217-afb8-feb2184fb974\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e77b1d-6c5d97c974-5r5f8" Oct 30 19:08:06.816485 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:06.816467 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kft5m\" (UniqueName: \"kubernetes.io/projected/56a9fce0-2511-4217-afb8-feb2184fb974-kube-api-access-kft5m\") pod \"maas-api-e2e-extmod-b-e77b1d-6c5d97c974-5r5f8\" (UID: \"56a9fce0-2511-4217-afb8-feb2184fb974\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e77b1d-6c5d97c974-5r5f8" Oct 30 19:08:06.816559 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:06.816525 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56a9fce0-2511-4217-afb8-feb2184fb974-maas-api-tls\") pod \"maas-api-e2e-extmod-b-e77b1d-6c5d97c974-5r5f8\" (UID: \"56a9fce0-2511-4217-afb8-feb2184fb974\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e77b1d-6c5d97c974-5r5f8" Oct 30 19:08:06.862861 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:06.862806 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:06.917444 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:06.917411 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56a9fce0-2511-4217-afb8-feb2184fb974-metrics-certs\") pod \"maas-api-e2e-extmod-b-e77b1d-6c5d97c974-5r5f8\" (UID: \"56a9fce0-2511-4217-afb8-feb2184fb974\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e77b1d-6c5d97c974-5r5f8" Oct 30 19:08:06.917631 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:06.917516 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kft5m\" (UniqueName: \"kubernetes.io/projected/56a9fce0-2511-4217-afb8-feb2184fb974-kube-api-access-kft5m\") pod \"maas-api-e2e-extmod-b-e77b1d-6c5d97c974-5r5f8\" (UID: \"56a9fce0-2511-4217-afb8-feb2184fb974\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e77b1d-6c5d97c974-5r5f8" Oct 30 19:08:06.917631 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:06.917585 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56a9fce0-2511-4217-afb8-feb2184fb974-maas-api-tls\") pod \"maas-api-e2e-extmod-b-e77b1d-6c5d97c974-5r5f8\" (UID: \"56a9fce0-2511-4217-afb8-feb2184fb974\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e77b1d-6c5d97c974-5r5f8" Oct 30 19:08:06.917749 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:06.917678 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56a9fce0-2511-4217-afb8-feb2184fb974-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-e77b1d-6c5d97c974-5r5f8\" (UID: \"56a9fce0-2511-4217-afb8-feb2184fb974\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e77b1d-6c5d97c974-5r5f8" Oct 30 19:08:06.918438 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:06.918412 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56a9fce0-2511-4217-afb8-feb2184fb974-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-e77b1d-6c5d97c974-5r5f8\" (UID: \"56a9fce0-2511-4217-afb8-feb2184fb974\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e77b1d-6c5d97c974-5r5f8" Oct 30 19:08:06.920243 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:06.920224 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56a9fce0-2511-4217-afb8-feb2184fb974-metrics-certs\") pod \"maas-api-e2e-extmod-b-e77b1d-6c5d97c974-5r5f8\" (UID: \"56a9fce0-2511-4217-afb8-feb2184fb974\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e77b1d-6c5d97c974-5r5f8" Oct 30 19:08:06.920320 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:06.920259 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56a9fce0-2511-4217-afb8-feb2184fb974-maas-api-tls\") pod \"maas-api-e2e-extmod-b-e77b1d-6c5d97c974-5r5f8\" (UID: \"56a9fce0-2511-4217-afb8-feb2184fb974\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e77b1d-6c5d97c974-5r5f8" Oct 30 19:08:06.929506 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:06.929481 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kft5m\" (UniqueName: \"kubernetes.io/projected/56a9fce0-2511-4217-afb8-feb2184fb974-kube-api-access-kft5m\") pod \"maas-api-e2e-extmod-b-e77b1d-6c5d97c974-5r5f8\" (UID: \"56a9fce0-2511-4217-afb8-feb2184fb974\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e77b1d-6c5d97c974-5r5f8" Oct 30 19:08:06.997866 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:06.997731 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:07.089989 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:07.089945 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e77b1d-6c5d97c974-5r5f8" Oct 30 19:08:07.378064 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:07.378033 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-e77b1d-6c5d97c974-5r5f8"] Oct 30 19:08:07.378369 ip-10-0-140-163 kubenswrapper[6960]: W1030 19:08:07.378348 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56a9fce0_2511_4217_afb8_feb2184fb974.slice/crio-2095306065e3c8931b0a70a30af6eedf917fe22f43b532e0f6c1c6f2da61d041 WatchSource:0}: Error finding container 2095306065e3c8931b0a70a30af6eedf917fe22f43b532e0f6c1c6f2da61d041: Status 404 returned error can't find the container with id 2095306065e3c8931b0a70a30af6eedf917fe22f43b532e0f6c1c6f2da61d041 Oct 30 19:08:07.503211 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:07.503109 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e77b1d-6c5d97c974-5r5f8" event={"ID":"56a9fce0-2511-4217-afb8-feb2184fb974","Type":"ContainerStarted","Data":"2095306065e3c8931b0a70a30af6eedf917fe22f43b532e0f6c1c6f2da61d041"} Oct 30 19:08:07.902568 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:07.902537 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:08.511579 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:08.511544 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e77b1d-6c5d97c974-5r5f8" event={"ID":"56a9fce0-2511-4217-afb8-feb2184fb974","Type":"ContainerStarted","Data":"a4528dde609c55db773c04d61851f2d6c6d8c43ba3112fd0e92f476621acf89c"} Oct 30 19:08:08.511801 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:08.511630 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e77b1d-6c5d97c974-5r5f8" Oct 30 19:08:08.535270 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:08.535214 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e77b1d-6c5d97c974-5r5f8" podStartSLOduration=2.052376582 podStartE2EDuration="2.535195807s" totalImagesPullingTime="482.819225ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:08:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:08:08.532018011 +0000 UTC m=+2056.061049151" watchObservedRunningTime="2026-10-30 19:08:08.535195807 +0000 UTC m=+2056.064226949" Oct 30 19:08:08.797191 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:08.797090 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:09.854654 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:09.854615 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:10.627689 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:10.627658 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:11.443102 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:11.443044 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:12.325775 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:12.325708 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:13.106662 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:13.106602 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:14.058755 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:14.058721 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:14.525913 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:14.525835 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e77b1d-6c5d97c974-5r5f8" Oct 30 19:08:14.900513 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:14.900480 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:15.667572 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:15.667534 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:16.523071 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:16.523036 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:17.350755 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:17.350706 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:18.147752 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:18.147713 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:19.136923 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:19.136888 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:19.915196 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:19.915107 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:20.720705 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:20.720645 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:21.545514 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:21.545475 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:22.533001 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:22.532966 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:23.337864 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:23.337831 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:23.933968 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:08:23.933934 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 19:08:24.188979 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:24.188862 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:25.001600 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:25.001564 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:25.747627 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:25.747592 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:26.618838 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:26.618784 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:27.471325 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:27.471294 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:28.422550 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:28.422517 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:29.209543 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:29.209452 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:30.068165 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:30.064786 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:31.114756 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:31.114722 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:32.005837 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:32.005800 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:33.025738 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:33.025698 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:33.138700 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:33.138664 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:34.015845 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:34.015765 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:34.322490 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:34.322461 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:34.440116 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:34.440078 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:34.752351 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:34.752257 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:34.961984 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:34.961931 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:35.148623 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:35.148583 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:35.300644 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:35.300608 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:35.932620 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:35.932579 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:36.051687 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:36.051635 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:36.459782 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:36.459748 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:36.584216 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:36.584133 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:36.803269 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:36.796965 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:36.928295 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:36.928248 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:37.052085 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:37.052054 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:37.205884 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:37.205848 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:38.046042 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:38.046007 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:38.178061 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:38.178011 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:38.425839 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:38.424881 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:38.860759 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:38.860716 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:39.061887 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:39.061856 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:39.186329 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:39.186244 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:39.774319 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:39.774267 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:39.883620 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:39.883578 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:40.610111 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:40.610073 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-748bfgcld"] Oct 30 19:08:40.615734 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:40.615711 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-748bfgcld" Oct 30 19:08:40.619818 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:40.619796 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-f7c6e3\"/\"kube-root-ca.crt\"" Oct 30 19:08:40.620419 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:40.620402 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-f7c6e3\"/\"agc-cred-5156225dde7c\"" Oct 30 19:08:40.620529 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:40.620462 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-f7c6e3\"/\"agc-cred-af4ab7befc3d\"" Oct 30 19:08:40.620603 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:40.620545 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-f7c6e3\"/\"agc-cred-e4e19c51892d\"" Oct 30 19:08:40.620757 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:40.620740 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-f7c6e3\"/\"payload-processing-external-model-plugins-e2e-extmod-b-f7c6e3\"" Oct 30 19:08:40.620935 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:40.620910 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-f7c6e3\"/\"payload-processing-external-model-e2e--342ea960-dockercfg-f6pzj\"" Oct 30 19:08:40.620935 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:40.620848 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-f7c6e3\"/\"routing-overlay\"" Oct 30 19:08:40.621184 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:40.621104 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-f7c6e3\"/\"openshift-service-ca.crt\"" Oct 30 19:08:40.625819 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:40.625796 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-748bfgcld"] Oct 30 19:08:40.716236 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:40.716202 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xm4gj\" (UniqueName: \"kubernetes.io/projected/459dcc0c-62d3-4f03-be1f-83985e46e965-kube-api-access-xm4gj\") pod \"payload-processing-external-model-e2e-extmod-b-f7c6e3-748bfgcld\" (UID: \"459dcc0c-62d3-4f03-be1f-83985e46e965\") " pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-748bfgcld" Oct 30 19:08:40.716395 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:40.716380 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/459dcc0c-62d3-4f03-be1f-83985e46e965-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f7c6e3-748bfgcld\" (UID: \"459dcc0c-62d3-4f03-be1f-83985e46e965\") " pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-748bfgcld" Oct 30 19:08:40.716448 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:40.716436 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/459dcc0c-62d3-4f03-be1f-83985e46e965-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f7c6e3-748bfgcld\" (UID: \"459dcc0c-62d3-4f03-be1f-83985e46e965\") " pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-748bfgcld" Oct 30 19:08:40.716533 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:40.716517 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/459dcc0c-62d3-4f03-be1f-83985e46e965-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-f7c6e3-748bfgcld\" (UID: \"459dcc0c-62d3-4f03-be1f-83985e46e965\") " pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-748bfgcld" Oct 30 19:08:40.832001 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:40.831955 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/459dcc0c-62d3-4f03-be1f-83985e46e965-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-f7c6e3-748bfgcld\" (UID: \"459dcc0c-62d3-4f03-be1f-83985e46e965\") " pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-748bfgcld" Oct 30 19:08:40.835231 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:40.835201 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xm4gj\" (UniqueName: \"kubernetes.io/projected/459dcc0c-62d3-4f03-be1f-83985e46e965-kube-api-access-xm4gj\") pod \"payload-processing-external-model-e2e-extmod-b-f7c6e3-748bfgcld\" (UID: \"459dcc0c-62d3-4f03-be1f-83985e46e965\") " pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-748bfgcld" Oct 30 19:08:40.840164 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:40.835963 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/459dcc0c-62d3-4f03-be1f-83985e46e965-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f7c6e3-748bfgcld\" (UID: \"459dcc0c-62d3-4f03-be1f-83985e46e965\") " pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-748bfgcld" Oct 30 19:08:40.840438 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:40.840417 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/459dcc0c-62d3-4f03-be1f-83985e46e965-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f7c6e3-748bfgcld\" (UID: \"459dcc0c-62d3-4f03-be1f-83985e46e965\") " pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-748bfgcld" Oct 30 19:08:40.854178 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:40.841128 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/459dcc0c-62d3-4f03-be1f-83985e46e965-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f7c6e3-748bfgcld\" (UID: \"459dcc0c-62d3-4f03-be1f-83985e46e965\") " pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-748bfgcld" Oct 30 19:08:40.855002 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:40.839762 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/459dcc0c-62d3-4f03-be1f-83985e46e965-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f7c6e3-748bfgcld\" (UID: \"459dcc0c-62d3-4f03-be1f-83985e46e965\") " pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-748bfgcld" Oct 30 19:08:40.865064 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:40.865001 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/459dcc0c-62d3-4f03-be1f-83985e46e965-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-f7c6e3-748bfgcld\" (UID: \"459dcc0c-62d3-4f03-be1f-83985e46e965\") " pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-748bfgcld" Oct 30 19:08:40.872952 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:40.872918 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xm4gj\" (UniqueName: \"kubernetes.io/projected/459dcc0c-62d3-4f03-be1f-83985e46e965-kube-api-access-xm4gj\") pod \"payload-processing-external-model-e2e-extmod-b-f7c6e3-748bfgcld\" (UID: \"459dcc0c-62d3-4f03-be1f-83985e46e965\") " pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-748bfgcld" Oct 30 19:08:40.951009 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:40.950977 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-748bfgcld" Oct 30 19:08:40.989787 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:40.989709 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:41.311929 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:41.311895 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-748bfgcld"] Oct 30 19:08:41.312500 ip-10-0-140-163 kubenswrapper[6960]: W1030 19:08:41.312474 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod459dcc0c_62d3_4f03_be1f_83985e46e965.slice/crio-2b7859817eade931ea816745115ed2470dfcbf4b82a7e034060c470ca6132ac4 WatchSource:0}: Error finding container 2b7859817eade931ea816745115ed2470dfcbf4b82a7e034060c470ca6132ac4: Status 404 returned error can't find the container with id 2b7859817eade931ea816745115ed2470dfcbf4b82a7e034060c470ca6132ac4 Oct 30 19:08:41.811043 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:41.811008 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-748bfgcld" event={"ID":"459dcc0c-62d3-4f03-be1f-83985e46e965","Type":"ContainerStarted","Data":"414475e3892dda6b7e8a1c7feda351af32bbee806c36428055df7e1be773029f"} Oct 30 19:08:41.811589 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:41.811076 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-748bfgcld" event={"ID":"459dcc0c-62d3-4f03-be1f-83985e46e965","Type":"ContainerStarted","Data":"2b7859817eade931ea816745115ed2470dfcbf4b82a7e034060c470ca6132ac4"} Oct 30 19:08:41.811589 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:41.811181 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-748bfgcld" Oct 30 19:08:41.831206 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:41.831159 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-748bfgcld" podStartSLOduration=1.831131965 podStartE2EDuration="1.831131965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:08:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:08:41.830306896 +0000 UTC m=+2089.359338044" watchObservedRunningTime="2026-10-30 19:08:41.831131965 +0000 UTC m=+2089.360163097" Oct 30 19:08:41.924152 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:41.924119 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:42.218123 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:42.218054 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-77bc87q7v"] Oct 30 19:08:42.222993 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:42.222974 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-77bc87q7v" Oct 30 19:08:42.227359 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:42.227336 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-4e94aa\"/\"payload-processing-external-model-plugins-e2e-extmod-b-4e94aa\"" Oct 30 19:08:42.230978 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:42.230956 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-4e94aa\"/\"agc-cred-a4629089eada\"" Oct 30 19:08:42.231085 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:42.231058 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-4e94aa\"/\"payload-processing-external-model-e2e--0d1ddd5a-dockercfg-7twb9\"" Oct 30 19:08:42.231716 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:42.231460 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-4e94aa\"/\"agc-cred-394aab6f23b5\"" Oct 30 19:08:42.232280 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:42.232256 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-4e94aa\"/\"agc-cred-ce5ac7999d53\"" Oct 30 19:08:42.232389 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:42.232298 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-4e94aa\"/\"kube-root-ca.crt\"" Oct 30 19:08:42.232936 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:42.232908 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-77bc87q7v"] Oct 30 19:08:42.239590 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:42.239572 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-4e94aa\"/\"openshift-service-ca.crt\"" Oct 30 19:08:42.239832 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:42.239819 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-4e94aa\"/\"routing-overlay\"" Oct 30 19:08:42.378634 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:42.378600 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ffac37d8-17e3-4a0f-b738-3ab400672fe5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4e94aa-77bc87q7v\" (UID: \"ffac37d8-17e3-4a0f-b738-3ab400672fe5\") " pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-77bc87q7v" Oct 30 19:08:42.378846 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:42.378831 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ffac37d8-17e3-4a0f-b738-3ab400672fe5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4e94aa-77bc87q7v\" (UID: \"ffac37d8-17e3-4a0f-b738-3ab400672fe5\") " pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-77bc87q7v" Oct 30 19:08:42.378900 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:42.378890 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffac37d8-17e3-4a0f-b738-3ab400672fe5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4e94aa-77bc87q7v\" (UID: \"ffac37d8-17e3-4a0f-b738-3ab400672fe5\") " pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-77bc87q7v" Oct 30 19:08:42.379006 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:42.378988 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzh6b\" (UniqueName: \"kubernetes.io/projected/ffac37d8-17e3-4a0f-b738-3ab400672fe5-kube-api-access-mzh6b\") pod \"payload-processing-external-model-e2e-extmod-b-4e94aa-77bc87q7v\" (UID: \"ffac37d8-17e3-4a0f-b738-3ab400672fe5\") " pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-77bc87q7v" Oct 30 19:08:42.483864 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:42.483645 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ffac37d8-17e3-4a0f-b738-3ab400672fe5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4e94aa-77bc87q7v\" (UID: \"ffac37d8-17e3-4a0f-b738-3ab400672fe5\") " pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-77bc87q7v" Oct 30 19:08:42.483864 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:42.483748 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffac37d8-17e3-4a0f-b738-3ab400672fe5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4e94aa-77bc87q7v\" (UID: \"ffac37d8-17e3-4a0f-b738-3ab400672fe5\") " pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-77bc87q7v" Oct 30 19:08:42.483864 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:42.483858 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzh6b\" (UniqueName: \"kubernetes.io/projected/ffac37d8-17e3-4a0f-b738-3ab400672fe5-kube-api-access-mzh6b\") pod \"payload-processing-external-model-e2e-extmod-b-4e94aa-77bc87q7v\" (UID: \"ffac37d8-17e3-4a0f-b738-3ab400672fe5\") " pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-77bc87q7v" Oct 30 19:08:42.484130 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:42.483953 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ffac37d8-17e3-4a0f-b738-3ab400672fe5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4e94aa-77bc87q7v\" (UID: \"ffac37d8-17e3-4a0f-b738-3ab400672fe5\") " pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-77bc87q7v" Oct 30 19:08:42.485084 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:42.485056 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ffac37d8-17e3-4a0f-b738-3ab400672fe5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4e94aa-77bc87q7v\" (UID: \"ffac37d8-17e3-4a0f-b738-3ab400672fe5\") " pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-77bc87q7v" Oct 30 19:08:42.486451 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:42.486386 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffac37d8-17e3-4a0f-b738-3ab400672fe5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4e94aa-77bc87q7v\" (UID: \"ffac37d8-17e3-4a0f-b738-3ab400672fe5\") " pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-77bc87q7v" Oct 30 19:08:42.497218 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:42.496184 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ffac37d8-17e3-4a0f-b738-3ab400672fe5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4e94aa-77bc87q7v\" (UID: \"ffac37d8-17e3-4a0f-b738-3ab400672fe5\") " pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-77bc87q7v" Oct 30 19:08:42.503983 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:42.503010 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzh6b\" (UniqueName: \"kubernetes.io/projected/ffac37d8-17e3-4a0f-b738-3ab400672fe5-kube-api-access-mzh6b\") pod \"payload-processing-external-model-e2e-extmod-b-4e94aa-77bc87q7v\" (UID: \"ffac37d8-17e3-4a0f-b738-3ab400672fe5\") " pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-77bc87q7v" Oct 30 19:08:42.541445 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:42.541411 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-77bc87q7v" Oct 30 19:08:42.778915 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:42.778811 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:42.875375 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:42.875351 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-77bc87q7v"] Oct 30 19:08:42.876215 ip-10-0-140-163 kubenswrapper[6960]: W1030 19:08:42.876135 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffac37d8_17e3_4a0f_b738_3ab400672fe5.slice/crio-87deec2c3d3bba470bdb0e2c7cb48c570119c8f049a90a2987b56daf143532c8 WatchSource:0}: Error finding container 87deec2c3d3bba470bdb0e2c7cb48c570119c8f049a90a2987b56daf143532c8: Status 404 returned error can't find the container with id 87deec2c3d3bba470bdb0e2c7cb48c570119c8f049a90a2987b56daf143532c8 Oct 30 19:08:43.567793 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:43.567717 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:43.686323 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:43.686283 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:43.858760 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:43.858660 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-77bc87q7v" event={"ID":"ffac37d8-17e3-4a0f-b738-3ab400672fe5","Type":"ContainerStarted","Data":"2469a732e40aff255f19b52ba1d64ac9929579831189a31968ba86a4264d93b0"} Oct 30 19:08:43.858760 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:43.858720 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-77bc87q7v" event={"ID":"ffac37d8-17e3-4a0f-b738-3ab400672fe5","Type":"ContainerStarted","Data":"87deec2c3d3bba470bdb0e2c7cb48c570119c8f049a90a2987b56daf143532c8"} Oct 30 19:08:43.858979 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:43.858857 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-77bc87q7v" Oct 30 19:08:43.890903 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:43.890847 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-77bc87q7v" podStartSLOduration=1.890832326 podStartE2EDuration="1.890832326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:08:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:08:43.885902049 +0000 UTC m=+2091.414933181" watchObservedRunningTime="2026-10-30 19:08:43.890832326 +0000 UTC m=+2091.419863463" Oct 30 19:08:44.248632 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:44.248538 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:44.361236 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:44.361201 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:44.937171 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:44.927451 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-77bc87q7v"] Oct 30 19:08:44.963524 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:08:44.961610 6960 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-4e94aa/agc-cred-a4629089eada: secret "agc-cred-a4629089eada" not found Oct 30 19:08:44.963524 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:08:44.961639 6960 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-77bc87q7v: secret "agc-cred-a4629089eada" not found Oct 30 19:08:44.963524 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:08:44.961711 6960 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ffac37d8-17e3-4a0f-b738-3ab400672fe5-provider-credentials podName:ffac37d8-17e3-4a0f-b738-3ab400672fe5 nodeName:}" failed. No retries permitted until 2026-10-30 19:08:45.461689675 +0000 UTC m=+2092.990720807 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/ffac37d8-17e3-4a0f-b738-3ab400672fe5-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-4e94aa-77bc87q7v" (UID: "ffac37d8-17e3-4a0f-b738-3ab400672fe5") : secret "agc-cred-a4629089eada" not found Oct 30 19:08:45.012173 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:45.005439 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:45.472221 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:08:45.472187 6960 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-4e94aa/agc-cred-a4629089eada: secret "agc-cred-a4629089eada" not found Oct 30 19:08:45.472461 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:08:45.472448 6960 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-77bc87q7v: secret "agc-cred-a4629089eada" not found Oct 30 19:08:45.472600 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:08:45.472588 6960 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ffac37d8-17e3-4a0f-b738-3ab400672fe5-provider-credentials podName:ffac37d8-17e3-4a0f-b738-3ab400672fe5 nodeName:}" failed. No retries permitted until 2026-10-30 19:08:46.472569031 +0000 UTC m=+2094.001600163 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/ffac37d8-17e3-4a0f-b738-3ab400672fe5-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-4e94aa-77bc87q7v" (UID: "ffac37d8-17e3-4a0f-b738-3ab400672fe5") : secret "agc-cred-a4629089eada" not found Oct 30 19:08:46.024848 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:46.024814 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:46.294589 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:46.294495 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:46.499429 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:46.499395 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:46.504101 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:08:46.504077 6960 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-4e94aa/agc-cred-a4629089eada: secret "agc-cred-a4629089eada" not found Oct 30 19:08:46.504101 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:08:46.504101 6960 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-77bc87q7v: secret "agc-cred-a4629089eada" not found Oct 30 19:08:46.504330 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:08:46.504180 6960 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ffac37d8-17e3-4a0f-b738-3ab400672fe5-provider-credentials podName:ffac37d8-17e3-4a0f-b738-3ab400672fe5 nodeName:}" failed. No retries permitted until 2026-10-30 19:08:48.504161528 +0000 UTC m=+2096.033192664 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/ffac37d8-17e3-4a0f-b738-3ab400672fe5-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-4e94aa-77bc87q7v" (UID: "ffac37d8-17e3-4a0f-b738-3ab400672fe5") : secret "agc-cred-a4629089eada" not found Oct 30 19:08:46.813214 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:46.813179 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:47.422493 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:47.422453 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:47.569931 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:47.569890 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:47.977707 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:47.977662 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:48.424538 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:48.424505 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:48.564731 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:08:48.564705 6960 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-4e94aa/agc-cred-a4629089eada: secret "agc-cred-a4629089eada" not found Oct 30 19:08:48.564731 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:08:48.564731 6960 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-77bc87q7v: secret "agc-cred-a4629089eada" not found Oct 30 19:08:48.564988 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:08:48.564794 6960 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ffac37d8-17e3-4a0f-b738-3ab400672fe5-provider-credentials podName:ffac37d8-17e3-4a0f-b738-3ab400672fe5 nodeName:}" failed. No retries permitted until 2026-10-30 19:08:52.564776236 +0000 UTC m=+2100.093807353 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/ffac37d8-17e3-4a0f-b738-3ab400672fe5-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-4e94aa-77bc87q7v" (UID: "ffac37d8-17e3-4a0f-b738-3ab400672fe5") : secret "agc-cred-a4629089eada" not found Oct 30 19:08:48.594487 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:48.594448 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:48.709133 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:48.709039 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:49.016897 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:49.016805 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:49.905034 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:49.904992 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:50.875795 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:50.875757 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-77bc87q7v" podUID="ffac37d8-17e3-4a0f-b738-3ab400672fe5" containerName="payload-processing" containerID="cri-o://2469a732e40aff255f19b52ba1d64ac9929579831189a31968ba86a4264d93b0" gracePeriod=25 Oct 30 19:08:51.003135 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:51.003103 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:51.224869 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:51.224841 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-77bc87q7v" Oct 30 19:08:51.313502 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:51.313467 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffac37d8-17e3-4a0f-b738-3ab400672fe5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffac37d8-17e3-4a0f-b738-3ab400672fe5-plugins-config-volume\") pod \"ffac37d8-17e3-4a0f-b738-3ab400672fe5\" (UID: \"ffac37d8-17e3-4a0f-b738-3ab400672fe5\") " Oct 30 19:08:51.313705 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:51.313556 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffac37d8-17e3-4a0f-b738-3ab400672fe5-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ffac37d8-17e3-4a0f-b738-3ab400672fe5-routing-overlay-volume\") pod \"ffac37d8-17e3-4a0f-b738-3ab400672fe5\" (UID: \"ffac37d8-17e3-4a0f-b738-3ab400672fe5\") " Oct 30 19:08:51.313705 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:51.313683 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffac37d8-17e3-4a0f-b738-3ab400672fe5-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ffac37d8-17e3-4a0f-b738-3ab400672fe5-provider-credentials\") pod \"ffac37d8-17e3-4a0f-b738-3ab400672fe5\" (UID: \"ffac37d8-17e3-4a0f-b738-3ab400672fe5\") " Oct 30 19:08:51.313833 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:51.313709 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffac37d8-17e3-4a0f-b738-3ab400672fe5-kube-api-access-mzh6b\" (UniqueName: \"kubernetes.io/projected/ffac37d8-17e3-4a0f-b738-3ab400672fe5-kube-api-access-mzh6b\") pod \"ffac37d8-17e3-4a0f-b738-3ab400672fe5\" (UID: \"ffac37d8-17e3-4a0f-b738-3ab400672fe5\") " Oct 30 19:08:51.313933 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:51.313909 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffac37d8-17e3-4a0f-b738-3ab400672fe5-routing-overlay-volume" pod "ffac37d8-17e3-4a0f-b738-3ab400672fe5" (UID: "ffac37d8-17e3-4a0f-b738-3ab400672fe5"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:08:51.313986 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:51.313936 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffac37d8-17e3-4a0f-b738-3ab400672fe5-plugins-config-volume" pod "ffac37d8-17e3-4a0f-b738-3ab400672fe5" (UID: "ffac37d8-17e3-4a0f-b738-3ab400672fe5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:08:51.314405 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:51.314384 6960 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffac37d8-17e3-4a0f-b738-3ab400672fe5-plugins-config-volume\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:08:51.314405 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:51.314404 6960 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ffac37d8-17e3-4a0f-b738-3ab400672fe5-routing-overlay-volume\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:08:51.316086 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:51.316060 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffac37d8-17e3-4a0f-b738-3ab400672fe5-provider-credentials" pod "ffac37d8-17e3-4a0f-b738-3ab400672fe5" (UID: "ffac37d8-17e3-4a0f-b738-3ab400672fe5"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:08:51.316193 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:51.316088 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffac37d8-17e3-4a0f-b738-3ab400672fe5-kube-api-access-mzh6b" pod "ffac37d8-17e3-4a0f-b738-3ab400672fe5" (UID: "ffac37d8-17e3-4a0f-b738-3ab400672fe5"). InnerVolumeSpecName "kube-api-access-mzh6b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:08:51.415840 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:51.415809 6960 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ffac37d8-17e3-4a0f-b738-3ab400672fe5-provider-credentials\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:08:51.415840 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:51.415835 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzh6b\" (UniqueName: \"kubernetes.io/projected/ffac37d8-17e3-4a0f-b738-3ab400672fe5-kube-api-access-mzh6b\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:08:51.922316 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:51.922283 6960 generic.go:356] "Generic (PLEG): container finished" podID="ffac37d8-17e3-4a0f-b738-3ab400672fe5" containerID="2469a732e40aff255f19b52ba1d64ac9929579831189a31968ba86a4264d93b0" exitCode=0 Oct 30 19:08:51.922526 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:51.922347 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-77bc87q7v" Oct 30 19:08:51.922526 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:51.922357 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-77bc87q7v" event={"ID":"ffac37d8-17e3-4a0f-b738-3ab400672fe5","Type":"ContainerDied","Data":"2469a732e40aff255f19b52ba1d64ac9929579831189a31968ba86a4264d93b0"} Oct 30 19:08:51.922526 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:51.922412 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-77bc87q7v" event={"ID":"ffac37d8-17e3-4a0f-b738-3ab400672fe5","Type":"ContainerDied","Data":"87deec2c3d3bba470bdb0e2c7cb48c570119c8f049a90a2987b56daf143532c8"} Oct 30 19:08:51.922526 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:51.922438 6960 scope.go:122] "RemoveContainer" containerID="2469a732e40aff255f19b52ba1d64ac9929579831189a31968ba86a4264d93b0" Oct 30 19:08:51.945678 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:51.945651 6960 scope.go:122] "RemoveContainer" containerID="2469a732e40aff255f19b52ba1d64ac9929579831189a31968ba86a4264d93b0" Oct 30 19:08:51.946048 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:08:51.946015 6960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2469a732e40aff255f19b52ba1d64ac9929579831189a31968ba86a4264d93b0\": container with ID starting with 2469a732e40aff255f19b52ba1d64ac9929579831189a31968ba86a4264d93b0 not found: ID does not exist" containerID="2469a732e40aff255f19b52ba1d64ac9929579831189a31968ba86a4264d93b0" Oct 30 19:08:51.946138 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:51.946065 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2469a732e40aff255f19b52ba1d64ac9929579831189a31968ba86a4264d93b0"} err="failed to get container status \"2469a732e40aff255f19b52ba1d64ac9929579831189a31968ba86a4264d93b0\": rpc error: code = NotFound desc = could not find container \"2469a732e40aff255f19b52ba1d64ac9929579831189a31968ba86a4264d93b0\": container with ID starting with 2469a732e40aff255f19b52ba1d64ac9929579831189a31968ba86a4264d93b0 not found: ID does not exist" Oct 30 19:08:51.961682 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:51.961653 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-77bc87q7v"] Oct 30 19:08:51.965728 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:51.965704 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-4e94aa/payload-processing-external-model-e2e-extmod-b-4e94aa-77bc87q7v"] Oct 30 19:08:52.346179 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:52.346088 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:52.596341 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:52.596258 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffac37d8-17e3-4a0f-b738-3ab400672fe5" path="/var/lib/kubelet/pods/ffac37d8-17e3-4a0f-b738-3ab400672fe5/volumes" Oct 30 19:08:52.853895 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:52.853820 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-748bfgcld" Oct 30 19:08:53.338472 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:53.338435 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:53.482694 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:53.482664 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:53.936227 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:08:53.936194 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 19:08:54.277098 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:54.277005 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:55.310590 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:55.310552 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:55.605578 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:55.605493 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:55.709857 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:55.709825 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:56.159301 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:56.159263 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:56.355943 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:56.355899 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:56.474500 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:56.474406 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:56.646966 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:56.646931 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:57.381409 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:57.381374 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-mswcj"] Oct 30 19:08:57.382505 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:57.382483 6960 cpu_manager.go:414] "containerMap: removing container" podUID="ffac37d8-17e3-4a0f-b738-3ab400672fe5" containerName="payload-processing" Oct 30 19:08:57.382590 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:57.382509 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffac37d8-17e3-4a0f-b738-3ab400672fe5" containerName="payload-processing" Oct 30 19:08:57.382743 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:57.382731 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffac37d8-17e3-4a0f-b738-3ab400672fe5" containerName="payload-processing" Oct 30 19:08:57.388237 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:57.388214 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-mswcj" Oct 30 19:08:57.402239 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:57.402215 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-f7c6e3-openshift-default-dockercfg-9qn52\"" Oct 30 19:08:57.407582 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:57.407547 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-mswcj"] Oct 30 19:08:57.571186 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:57.552166 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2fbe8d6-d7b0-4814-a665-e47be1a45176-credential-socket\") pod \"e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-mswcj\" (UID: \"f2fbe8d6-d7b0-4814-a665-e47be1a45176\") " pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-mswcj" Oct 30 19:08:57.571186 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:57.552285 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2fbe8d6-d7b0-4814-a665-e47be1a45176-istio-data\") pod \"e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-mswcj\" (UID: \"f2fbe8d6-d7b0-4814-a665-e47be1a45176\") " pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-mswcj" Oct 30 19:08:57.571186 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:57.552340 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n79rg\" (UniqueName: \"kubernetes.io/projected/f2fbe8d6-d7b0-4814-a665-e47be1a45176-kube-api-access-n79rg\") pod \"e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-mswcj\" (UID: \"f2fbe8d6-d7b0-4814-a665-e47be1a45176\") " pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-mswcj" Oct 30 19:08:57.571186 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:57.552393 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2fbe8d6-d7b0-4814-a665-e47be1a45176-workload-certs\") pod \"e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-mswcj\" (UID: \"f2fbe8d6-d7b0-4814-a665-e47be1a45176\") " pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-mswcj" Oct 30 19:08:57.571186 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:57.552466 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2fbe8d6-d7b0-4814-a665-e47be1a45176-istio-envoy\") pod \"e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-mswcj\" (UID: \"f2fbe8d6-d7b0-4814-a665-e47be1a45176\") " pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-mswcj" Oct 30 19:08:57.571186 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:57.552514 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2fbe8d6-d7b0-4814-a665-e47be1a45176-istio-token\") pod \"e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-mswcj\" (UID: \"f2fbe8d6-d7b0-4814-a665-e47be1a45176\") " pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-mswcj" Oct 30 19:08:57.571186 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:57.552641 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2fbe8d6-d7b0-4814-a665-e47be1a45176-workload-socket\") pod \"e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-mswcj\" (UID: \"f2fbe8d6-d7b0-4814-a665-e47be1a45176\") " pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-mswcj" Oct 30 19:08:57.571186 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:57.552675 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2fbe8d6-d7b0-4814-a665-e47be1a45176-istiod-ca-cert\") pod \"e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-mswcj\" (UID: \"f2fbe8d6-d7b0-4814-a665-e47be1a45176\") " pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-mswcj" Oct 30 19:08:57.571186 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:57.552724 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2fbe8d6-d7b0-4814-a665-e47be1a45176-istio-podinfo\") pod \"e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-mswcj\" (UID: \"f2fbe8d6-d7b0-4814-a665-e47be1a45176\") " pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-mswcj" Oct 30 19:08:57.658215 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:57.655757 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2fbe8d6-d7b0-4814-a665-e47be1a45176-credential-socket\") pod \"e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-mswcj\" (UID: \"f2fbe8d6-d7b0-4814-a665-e47be1a45176\") " pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-mswcj" Oct 30 19:08:57.658215 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:57.655890 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2fbe8d6-d7b0-4814-a665-e47be1a45176-istio-data\") pod \"e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-mswcj\" (UID: \"f2fbe8d6-d7b0-4814-a665-e47be1a45176\") " pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-mswcj" Oct 30 19:08:57.658215 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:57.655948 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n79rg\" (UniqueName: \"kubernetes.io/projected/f2fbe8d6-d7b0-4814-a665-e47be1a45176-kube-api-access-n79rg\") pod \"e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-mswcj\" (UID: \"f2fbe8d6-d7b0-4814-a665-e47be1a45176\") " pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-mswcj" Oct 30 19:08:57.658215 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:57.656020 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2fbe8d6-d7b0-4814-a665-e47be1a45176-workload-certs\") pod \"e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-mswcj\" (UID: \"f2fbe8d6-d7b0-4814-a665-e47be1a45176\") " pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-mswcj" Oct 30 19:08:57.658215 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:57.656550 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2fbe8d6-d7b0-4814-a665-e47be1a45176-istio-data\") pod \"e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-mswcj\" (UID: \"f2fbe8d6-d7b0-4814-a665-e47be1a45176\") " pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-mswcj" Oct 30 19:08:57.658215 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:57.657212 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2fbe8d6-d7b0-4814-a665-e47be1a45176-credential-socket\") pod \"e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-mswcj\" (UID: \"f2fbe8d6-d7b0-4814-a665-e47be1a45176\") " pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-mswcj" Oct 30 19:08:57.658741 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:57.658224 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2fbe8d6-d7b0-4814-a665-e47be1a45176-workload-certs\") pod \"e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-mswcj\" (UID: \"f2fbe8d6-d7b0-4814-a665-e47be1a45176\") " pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-mswcj" Oct 30 19:08:57.665176 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:57.656114 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2fbe8d6-d7b0-4814-a665-e47be1a45176-istio-envoy\") pod \"e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-mswcj\" (UID: \"f2fbe8d6-d7b0-4814-a665-e47be1a45176\") " pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-mswcj" Oct 30 19:08:57.665176 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:57.664634 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2fbe8d6-d7b0-4814-a665-e47be1a45176-istio-token\") pod \"e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-mswcj\" (UID: \"f2fbe8d6-d7b0-4814-a665-e47be1a45176\") " pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-mswcj" Oct 30 19:08:57.665176 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:57.664876 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2fbe8d6-d7b0-4814-a665-e47be1a45176-workload-socket\") pod \"e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-mswcj\" (UID: \"f2fbe8d6-d7b0-4814-a665-e47be1a45176\") " pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-mswcj" Oct 30 19:08:57.665176 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:57.664949 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2fbe8d6-d7b0-4814-a665-e47be1a45176-istiod-ca-cert\") pod \"e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-mswcj\" (UID: \"f2fbe8d6-d7b0-4814-a665-e47be1a45176\") " pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-mswcj" Oct 30 19:08:57.665176 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:57.665007 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2fbe8d6-d7b0-4814-a665-e47be1a45176-istio-podinfo\") pod \"e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-mswcj\" (UID: \"f2fbe8d6-d7b0-4814-a665-e47be1a45176\") " pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-mswcj" Oct 30 19:08:57.673170 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:57.667762 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2fbe8d6-d7b0-4814-a665-e47be1a45176-workload-socket\") pod \"e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-mswcj\" (UID: \"f2fbe8d6-d7b0-4814-a665-e47be1a45176\") " pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-mswcj" Oct 30 19:08:57.673170 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:57.668390 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2fbe8d6-d7b0-4814-a665-e47be1a45176-istiod-ca-cert\") pod \"e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-mswcj\" (UID: \"f2fbe8d6-d7b0-4814-a665-e47be1a45176\") " pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-mswcj" Oct 30 19:08:57.682976 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:57.682910 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2fbe8d6-d7b0-4814-a665-e47be1a45176-istio-envoy\") pod \"e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-mswcj\" (UID: \"f2fbe8d6-d7b0-4814-a665-e47be1a45176\") " pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-mswcj" Oct 30 19:08:57.691189 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:57.687383 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2fbe8d6-d7b0-4814-a665-e47be1a45176-istio-podinfo\") pod \"e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-mswcj\" (UID: \"f2fbe8d6-d7b0-4814-a665-e47be1a45176\") " pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-mswcj" Oct 30 19:08:57.691189 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:57.690893 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n79rg\" (UniqueName: \"kubernetes.io/projected/f2fbe8d6-d7b0-4814-a665-e47be1a45176-kube-api-access-n79rg\") pod \"e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-mswcj\" (UID: \"f2fbe8d6-d7b0-4814-a665-e47be1a45176\") " pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-mswcj" Oct 30 19:08:57.691933 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:57.691893 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2fbe8d6-d7b0-4814-a665-e47be1a45176-istio-token\") pod \"e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-mswcj\" (UID: \"f2fbe8d6-d7b0-4814-a665-e47be1a45176\") " pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-mswcj" Oct 30 19:08:57.703877 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:57.703851 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-mswcj" Oct 30 19:08:57.747698 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:57.747670 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:57.867436 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:57.858764 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:58.063608 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:58.059677 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-mswcj"] Oct 30 19:08:58.073983 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:58.073947 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 19:08:58.074126 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:58.074036 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 19:08:58.074126 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:58.074080 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 19:08:58.928661 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:58.928627 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:08:58.991704 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:58.987865 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-mswcj" event={"ID":"f2fbe8d6-d7b0-4814-a665-e47be1a45176","Type":"ContainerStarted","Data":"0a85f9560a19dfd707d65ce74e658762b4f167dc607fd4ec82053e2debd1b265"} Oct 30 19:08:58.991704 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:58.987934 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-mswcj" event={"ID":"f2fbe8d6-d7b0-4814-a665-e47be1a45176","Type":"ContainerStarted","Data":"7b305c48c5f27da40aaaceda012848a083f35fee0b75526ac9e5c1d973a3232d"} Oct 30 19:08:59.028753 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:59.020329 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-mswcj" podStartSLOduration=2.020291762 podStartE2EDuration="2.020291762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:08:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:08:59.014312361 +0000 UTC m=+2106.543343522" watchObservedRunningTime="2026-10-30 19:08:59.020291762 +0000 UTC m=+2106.549322900" Oct 30 19:08:59.704351 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:59.704309 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-mswcj" Oct 30 19:08:59.720781 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:59.720742 6960 patch_prober.go:29] interesting pod/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-mswcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 30 19:08:59.720957 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:08:59.720806 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-mswcj" podUID="f2fbe8d6-d7b0-4814-a665-e47be1a45176" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 30 19:09:00.011137 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:00.011044 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:00.705226 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:00.705191 6960 patch_prober.go:29] interesting pod/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-mswcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 30 19:09:00.705417 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:00.705261 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-mswcj" podUID="f2fbe8d6-d7b0-4814-a665-e47be1a45176" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 30 19:09:01.321362 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:01.321320 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:01.704936 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:01.704860 6960 patch_prober.go:29] interesting pod/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-mswcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 30 19:09:01.704936 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:01.704923 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-mswcj" podUID="f2fbe8d6-d7b0-4814-a665-e47be1a45176" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 30 19:09:02.240647 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:02.240608 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:02.705019 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:02.704982 6960 patch_prober.go:29] interesting pod/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-mswcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 30 19:09:02.705472 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:02.705052 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-mswcj" podUID="f2fbe8d6-d7b0-4814-a665-e47be1a45176" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 30 19:09:03.529170 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:03.528957 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:03.704408 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:03.704372 6960 patch_prober.go:29] interesting pod/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-mswcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 30 19:09:03.704590 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:03.704418 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-mswcj" podUID="f2fbe8d6-d7b0-4814-a665-e47be1a45176" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 30 19:09:03.720843 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:03.720706 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:03.889558 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:03.889506 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:04.208604 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:04.208514 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:04.704473 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:04.704434 6960 patch_prober.go:29] interesting pod/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-mswcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 30 19:09:04.704690 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:04.704498 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-mswcj" podUID="f2fbe8d6-d7b0-4814-a665-e47be1a45176" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 30 19:09:04.775830 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:04.775789 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:05.655820 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:05.655783 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:05.717768 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:05.717724 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-mswcj" Oct 30 19:09:05.718202 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:05.718184 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-mswcj" Oct 30 19:09:05.719518 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:05.719495 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-mswcj" Oct 30 19:09:06.620791 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:06.620756 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:07.557053 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:07.557017 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:08.362935 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:08.362896 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:09.247861 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:09.247826 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:10.031823 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:10.031789 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:10.947875 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:10.947835 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:11.314409 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:11.314383 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:11.445755 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:11.445723 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:11.657437 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:11.657329 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:12.017124 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:12.017029 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:12.468871 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:12.468821 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:12.754993 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:12.754911 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:13.617226 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:13.617189 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:14.420021 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:14.419982 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:15.194104 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:15.194069 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:16.319631 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:16.319596 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:17.143254 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:17.143219 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:17.903687 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:17.903653 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:18.720480 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:18.720439 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:19.551782 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:19.550892 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:20.436938 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:20.436900 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:21.583194 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:21.583105 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:22.749852 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:22.749813 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:23.771511 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:23.771445 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:24.021162 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:24.015336 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:24.089432 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:09:24.089398 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 19:09:24.280744 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:24.280713 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:24.688181 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:24.688111 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:25.006997 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:25.006920 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:25.147687 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:25.147654 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:25.698699 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:25.698652 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:25.846000 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:25.845959 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:25.963098 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:25.963003 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:27.068991 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:27.068957 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:27.266715 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:27.266672 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:28.406977 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:28.406939 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:29.385187 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:29.385135 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:30.306081 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:30.306050 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:31.099502 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:31.099466 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:32.027451 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:32.027409 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:33.013194 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:33.013121 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:33.142229 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:33.142189 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:33.786993 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:33.786960 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:34.622312 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:34.622280 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:35.511292 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:35.511256 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:36.326350 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:36.326310 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:37.209847 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:37.209810 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:38.088180 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:38.086187 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:38.364417 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:38.364329 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:38.475017 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:38.474984 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:39.386758 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:39.386719 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:40.016693 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:40.016664 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-748bfgcld"] Oct 30 19:09:40.041613 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:40.041581 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:40.609579 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:40.609531 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:41.493332 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:41.493295 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:42.320085 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:42.320051 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:43.128131 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:43.126121 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:44.164850 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:44.160865 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:45.017198 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:45.017118 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-748bfgcld" podUID="459dcc0c-62d3-4f03-be1f-83985e46e965" containerName="payload-processing" containerID="cri-o://414475e3892dda6b7e8a1c7feda351af32bbee806c36428055df7e1be773029f" gracePeriod=25 Oct 30 19:09:45.236460 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:45.236378 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:45.487293 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:45.487270 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-748bfgcld" Oct 30 19:09:45.530101 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:45.530066 6960 generic.go:356] "Generic (PLEG): container finished" podID="459dcc0c-62d3-4f03-be1f-83985e46e965" containerID="414475e3892dda6b7e8a1c7feda351af32bbee806c36428055df7e1be773029f" exitCode=0 Oct 30 19:09:45.530313 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:45.530201 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-748bfgcld" event={"ID":"459dcc0c-62d3-4f03-be1f-83985e46e965","Type":"ContainerDied","Data":"414475e3892dda6b7e8a1c7feda351af32bbee806c36428055df7e1be773029f"} Oct 30 19:09:45.530313 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:45.530240 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-748bfgcld" Oct 30 19:09:45.530313 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:45.530254 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-748bfgcld" event={"ID":"459dcc0c-62d3-4f03-be1f-83985e46e965","Type":"ContainerDied","Data":"2b7859817eade931ea816745115ed2470dfcbf4b82a7e034060c470ca6132ac4"} Oct 30 19:09:45.530313 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:45.530277 6960 scope.go:122] "RemoveContainer" containerID="414475e3892dda6b7e8a1c7feda351af32bbee806c36428055df7e1be773029f" Oct 30 19:09:45.543994 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:45.543977 6960 scope.go:122] "RemoveContainer" containerID="414475e3892dda6b7e8a1c7feda351af32bbee806c36428055df7e1be773029f" Oct 30 19:09:45.544348 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:09:45.544325 6960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"414475e3892dda6b7e8a1c7feda351af32bbee806c36428055df7e1be773029f\": container with ID starting with 414475e3892dda6b7e8a1c7feda351af32bbee806c36428055df7e1be773029f not found: ID does not exist" containerID="414475e3892dda6b7e8a1c7feda351af32bbee806c36428055df7e1be773029f" Oct 30 19:09:45.544485 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:45.544352 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"414475e3892dda6b7e8a1c7feda351af32bbee806c36428055df7e1be773029f"} err="failed to get container status \"414475e3892dda6b7e8a1c7feda351af32bbee806c36428055df7e1be773029f\": rpc error: code = NotFound desc = could not find container \"414475e3892dda6b7e8a1c7feda351af32bbee806c36428055df7e1be773029f\": container with ID starting with 414475e3892dda6b7e8a1c7feda351af32bbee806c36428055df7e1be773029f not found: ID does not exist" Oct 30 19:09:45.591445 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:45.591418 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/459dcc0c-62d3-4f03-be1f-83985e46e965-kube-api-access-xm4gj\" (UniqueName: \"kubernetes.io/projected/459dcc0c-62d3-4f03-be1f-83985e46e965-kube-api-access-xm4gj\") pod \"459dcc0c-62d3-4f03-be1f-83985e46e965\" (UID: \"459dcc0c-62d3-4f03-be1f-83985e46e965\") " Oct 30 19:09:45.591578 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:45.591456 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/459dcc0c-62d3-4f03-be1f-83985e46e965-provider-credentials\" (UniqueName: \"kubernetes.io/projected/459dcc0c-62d3-4f03-be1f-83985e46e965-provider-credentials\") pod \"459dcc0c-62d3-4f03-be1f-83985e46e965\" (UID: \"459dcc0c-62d3-4f03-be1f-83985e46e965\") " Oct 30 19:09:45.591578 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:45.591514 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/459dcc0c-62d3-4f03-be1f-83985e46e965-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/459dcc0c-62d3-4f03-be1f-83985e46e965-plugins-config-volume\") pod \"459dcc0c-62d3-4f03-be1f-83985e46e965\" (UID: \"459dcc0c-62d3-4f03-be1f-83985e46e965\") " Oct 30 19:09:45.591662 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:45.591634 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/459dcc0c-62d3-4f03-be1f-83985e46e965-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/459dcc0c-62d3-4f03-be1f-83985e46e965-routing-overlay-volume\") pod \"459dcc0c-62d3-4f03-be1f-83985e46e965\" (UID: \"459dcc0c-62d3-4f03-be1f-83985e46e965\") " Oct 30 19:09:45.592049 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:45.591977 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/459dcc0c-62d3-4f03-be1f-83985e46e965-plugins-config-volume" pod "459dcc0c-62d3-4f03-be1f-83985e46e965" (UID: "459dcc0c-62d3-4f03-be1f-83985e46e965"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:09:45.592049 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:45.591974 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/459dcc0c-62d3-4f03-be1f-83985e46e965-routing-overlay-volume" pod "459dcc0c-62d3-4f03-be1f-83985e46e965" (UID: "459dcc0c-62d3-4f03-be1f-83985e46e965"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:09:45.592338 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:45.592136 6960 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/459dcc0c-62d3-4f03-be1f-83985e46e965-plugins-config-volume\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:09:45.592338 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:45.592174 6960 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/459dcc0c-62d3-4f03-be1f-83985e46e965-routing-overlay-volume\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:09:45.593795 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:45.593774 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/459dcc0c-62d3-4f03-be1f-83985e46e965-provider-credentials" pod "459dcc0c-62d3-4f03-be1f-83985e46e965" (UID: "459dcc0c-62d3-4f03-be1f-83985e46e965"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:09:45.593881 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:45.593803 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/459dcc0c-62d3-4f03-be1f-83985e46e965-kube-api-access-xm4gj" pod "459dcc0c-62d3-4f03-be1f-83985e46e965" (UID: "459dcc0c-62d3-4f03-be1f-83985e46e965"). InnerVolumeSpecName "kube-api-access-xm4gj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:09:45.696791 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:45.696042 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xm4gj\" (UniqueName: \"kubernetes.io/projected/459dcc0c-62d3-4f03-be1f-83985e46e965-kube-api-access-xm4gj\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:09:45.700454 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:45.700423 6960 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/459dcc0c-62d3-4f03-be1f-83985e46e965-provider-credentials\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:09:45.896291 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:45.896198 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-748bfgcld"] Oct 30 19:09:45.906167 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:45.906118 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-748bfgcld"] Oct 30 19:09:46.232687 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:46.232595 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:46.438815 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:46.438774 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:46.546744 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:46.546648 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:46.594513 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:46.594487 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="459dcc0c-62d3-4f03-be1f-83985e46e965" path="/var/lib/kubelet/pods/459dcc0c-62d3-4f03-be1f-83985e46e965/volumes" Oct 30 19:09:46.653812 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:46.653779 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:47.175452 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:47.175413 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:48.112830 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:48.112794 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:49.135634 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:49.135592 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:49.924643 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:49.924614 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:50.810755 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:50.810722 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:51.732396 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:51.732364 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:52.485984 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:52.485828 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:52.601911 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:52.601875 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:52.694168 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:52.677495 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-6cdfmd5lg"] Oct 30 19:09:52.694168 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:52.678691 6960 cpu_manager.go:414] "containerMap: removing container" podUID="459dcc0c-62d3-4f03-be1f-83985e46e965" containerName="payload-processing" Oct 30 19:09:52.694168 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:52.678714 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="459dcc0c-62d3-4f03-be1f-83985e46e965" containerName="payload-processing" Oct 30 19:09:52.694168 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:52.678984 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="459dcc0c-62d3-4f03-be1f-83985e46e965" containerName="payload-processing" Oct 30 19:09:52.706179 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:52.699758 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-6cdfmd5lg" Oct 30 19:09:52.716745 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:52.716718 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-f7c6e3\"/\"payload-processing-external-model-e2e--342ea960-dockercfg-f6pzj\"" Oct 30 19:09:52.717045 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:52.717021 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-f7c6e3\"/\"agc-cred-5156225dde7c\"" Oct 30 19:09:52.720787 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:52.717619 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-f7c6e3\"/\"kube-root-ca.crt\"" Oct 30 19:09:52.720787 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:52.718027 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-f7c6e3\"/\"payload-processing-external-model-plugins-e2e-extmod-b-f7c6e3\"" Oct 30 19:09:52.720787 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:52.718071 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-f7c6e3\"/\"routing-overlay\"" Oct 30 19:09:52.720787 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:52.718130 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-f7c6e3\"/\"openshift-service-ca.crt\"" Oct 30 19:09:52.807321 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:52.807283 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-6cdfmd5lg"] Oct 30 19:09:52.886641 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:52.884761 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c5d059ac-6de0-4bac-a747-c11c684a2ceb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-f7c6e3-6cdfmd5lg\" (UID: \"c5d059ac-6de0-4bac-a747-c11c684a2ceb\") " pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-6cdfmd5lg" Oct 30 19:09:52.886641 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:52.884902 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c5d059ac-6de0-4bac-a747-c11c684a2ceb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f7c6e3-6cdfmd5lg\" (UID: \"c5d059ac-6de0-4bac-a747-c11c684a2ceb\") " pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-6cdfmd5lg" Oct 30 19:09:52.886641 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:52.885111 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5d059ac-6de0-4bac-a747-c11c684a2ceb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f7c6e3-6cdfmd5lg\" (UID: \"c5d059ac-6de0-4bac-a747-c11c684a2ceb\") " pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-6cdfmd5lg" Oct 30 19:09:52.886641 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:52.885161 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stdcq\" (UniqueName: \"kubernetes.io/projected/c5d059ac-6de0-4bac-a747-c11c684a2ceb-kube-api-access-stdcq\") pod \"payload-processing-external-model-e2e-extmod-b-f7c6e3-6cdfmd5lg\" (UID: \"c5d059ac-6de0-4bac-a747-c11c684a2ceb\") " pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-6cdfmd5lg" Oct 30 19:09:52.987265 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:52.987203 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c5d059ac-6de0-4bac-a747-c11c684a2ceb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-f7c6e3-6cdfmd5lg\" (UID: \"c5d059ac-6de0-4bac-a747-c11c684a2ceb\") " pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-6cdfmd5lg" Oct 30 19:09:52.987577 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:52.987335 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c5d059ac-6de0-4bac-a747-c11c684a2ceb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f7c6e3-6cdfmd5lg\" (UID: \"c5d059ac-6de0-4bac-a747-c11c684a2ceb\") " pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-6cdfmd5lg" Oct 30 19:09:52.987577 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:52.987526 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5d059ac-6de0-4bac-a747-c11c684a2ceb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f7c6e3-6cdfmd5lg\" (UID: \"c5d059ac-6de0-4bac-a747-c11c684a2ceb\") " pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-6cdfmd5lg" Oct 30 19:09:52.987577 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:52.987559 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stdcq\" (UniqueName: \"kubernetes.io/projected/c5d059ac-6de0-4bac-a747-c11c684a2ceb-kube-api-access-stdcq\") pod \"payload-processing-external-model-e2e-extmod-b-f7c6e3-6cdfmd5lg\" (UID: \"c5d059ac-6de0-4bac-a747-c11c684a2ceb\") " pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-6cdfmd5lg" Oct 30 19:09:52.988259 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:52.988234 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c5d059ac-6de0-4bac-a747-c11c684a2ceb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f7c6e3-6cdfmd5lg\" (UID: \"c5d059ac-6de0-4bac-a747-c11c684a2ceb\") " pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-6cdfmd5lg" Oct 30 19:09:52.988494 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:52.988473 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5d059ac-6de0-4bac-a747-c11c684a2ceb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f7c6e3-6cdfmd5lg\" (UID: \"c5d059ac-6de0-4bac-a747-c11c684a2ceb\") " pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-6cdfmd5lg" Oct 30 19:09:52.990209 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:52.990184 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c5d059ac-6de0-4bac-a747-c11c684a2ceb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-f7c6e3-6cdfmd5lg\" (UID: \"c5d059ac-6de0-4bac-a747-c11c684a2ceb\") " pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-6cdfmd5lg" Oct 30 19:09:53.000461 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:53.000438 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stdcq\" (UniqueName: \"kubernetes.io/projected/c5d059ac-6de0-4bac-a747-c11c684a2ceb-kube-api-access-stdcq\") pod \"payload-processing-external-model-e2e-extmod-b-f7c6e3-6cdfmd5lg\" (UID: \"c5d059ac-6de0-4bac-a747-c11c684a2ceb\") " pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-6cdfmd5lg" Oct 30 19:09:53.075120 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:53.075034 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:53.213394 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:53.213355 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-6cdfmd5lg" Oct 30 19:09:53.503056 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:53.503021 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:53.531836 ip-10-0-140-163 kubenswrapper[6960]: W1030 19:09:53.531803 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5d059ac_6de0_4bac_a747_c11c684a2ceb.slice/crio-363c6ae54ceefa332601255f84228130c1eb7b7dcdecc19bf7d3c4fd252a2571 WatchSource:0}: Error finding container 363c6ae54ceefa332601255f84228130c1eb7b7dcdecc19bf7d3c4fd252a2571: Status 404 returned error can't find the container with id 363c6ae54ceefa332601255f84228130c1eb7b7dcdecc19bf7d3c4fd252a2571 Oct 30 19:09:53.531973 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:53.531865 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-6cdfmd5lg"] Oct 30 19:09:53.594455 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:53.594424 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-6cdfmd5lg" event={"ID":"c5d059ac-6de0-4bac-a747-c11c684a2ceb","Type":"ContainerStarted","Data":"363c6ae54ceefa332601255f84228130c1eb7b7dcdecc19bf7d3c4fd252a2571"} Oct 30 19:09:53.933865 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:09:53.933828 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 19:09:54.128003 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:54.127964 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:54.265485 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:54.265405 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:54.602610 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:54.602575 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-6cdfmd5lg" event={"ID":"c5d059ac-6de0-4bac-a747-c11c684a2ceb","Type":"ContainerStarted","Data":"bb6594b0d3780a870f441df18da437752a0299f0bf6bb2e1bcb7f943bc76c135"} Oct 30 19:09:54.603107 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:54.602629 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-6cdfmd5lg" Oct 30 19:09:54.623118 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:54.623070 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-6cdfmd5lg" podStartSLOduration=2.623056511 podStartE2EDuration="2.623056511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:09:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:09:54.620740714 +0000 UTC m=+2162.149771852" watchObservedRunningTime="2026-10-30 19:09:54.623056511 +0000 UTC m=+2162.152087648" Oct 30 19:09:55.037792 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:55.037696 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:55.159537 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:55.157814 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:55.906877 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:55.906833 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:56.662373 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:56.662340 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:57.521809 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:57.521770 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:58.346085 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:58.346047 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:09:59.174274 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:09:59.174236 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:00.096981 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:00.096942 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:00.824250 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:00.824216 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:00.947562 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:00.947528 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:01.081800 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:01.081718 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:01.934016 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:01.933979 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:02.861794 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:02.861758 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:03.733312 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:03.733228 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:04.303594 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:04.303557 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:04.415578 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:04.415547 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:04.567751 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:04.567714 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:05.435216 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:05.435183 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:05.613520 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:05.613493 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-6cdfmd5lg" Oct 30 19:10:06.312934 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:06.312899 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:07.145713 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:07.145677 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:08.001741 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:08.001705 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:08.960632 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:08.960596 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:09.742312 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:09.742276 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:10.614911 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:10.614874 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:11.609152 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:11.609111 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:12.380756 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:12.380721 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:13.233598 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:13.233568 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:14.101950 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:14.101914 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:14.904373 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:14.904338 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:15.921799 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:15.921761 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:16.869077 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:16.869038 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:17.863805 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:17.863751 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:18.661797 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:18.661763 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:19.489343 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:19.489307 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:20.783392 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:20.783359 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:21.633215 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:21.633183 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:22.413258 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:22.413221 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:23.189886 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:23.189826 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:23.933313 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:10:23.933282 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 19:10:24.477532 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:24.477497 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:25.456908 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:25.456874 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:26.230735 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:26.230686 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:27.141055 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:27.141023 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:27.914761 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:27.914723 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:28.715379 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:28.715333 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:29.525732 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:29.525696 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:30.319375 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:30.319340 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:31.095293 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:31.095255 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:31.263966 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:31.263928 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:32.015239 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:32.015179 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:32.814255 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:32.814222 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:33.611674 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:33.611635 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:34.432427 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:34.432383 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:35.390207 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:35.390174 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:35.574736 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:35.574705 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-6cdfmd5lg"] Oct 30 19:10:35.598736 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:35.598699 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:35.716055 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:35.715964 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:36.235195 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:36.235156 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:36.446046 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:36.446006 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:36.563201 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:36.563169 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:36.998435 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:36.998358 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:38.026479 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:38.026438 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:38.941518 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:38.941475 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:39.047697 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:39.047661 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:40.105165 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:40.102121 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:40.218457 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:40.218377 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:40.333906 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:40.333872 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:40.577960 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:40.577918 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-6cdfmd5lg" podUID="c5d059ac-6de0-4bac-a747-c11c684a2ceb" containerName="payload-processing" containerID="cri-o://bb6594b0d3780a870f441df18da437752a0299f0bf6bb2e1bcb7f943bc76c135" gracePeriod=25 Oct 30 19:10:40.720947 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:40.720910 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:40.839230 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:40.837658 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:40.888678 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:40.888653 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-6cdfmd5lg" Oct 30 19:10:40.955113 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:40.955084 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5d059ac-6de0-4bac-a747-c11c684a2ceb-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c5d059ac-6de0-4bac-a747-c11c684a2ceb-provider-credentials\") pod \"c5d059ac-6de0-4bac-a747-c11c684a2ceb\" (UID: \"c5d059ac-6de0-4bac-a747-c11c684a2ceb\") " Oct 30 19:10:40.955336 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:40.955189 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5d059ac-6de0-4bac-a747-c11c684a2ceb-kube-api-access-stdcq\" (UniqueName: \"kubernetes.io/projected/c5d059ac-6de0-4bac-a747-c11c684a2ceb-kube-api-access-stdcq\") pod \"c5d059ac-6de0-4bac-a747-c11c684a2ceb\" (UID: \"c5d059ac-6de0-4bac-a747-c11c684a2ceb\") " Oct 30 19:10:40.955336 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:40.955229 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5d059ac-6de0-4bac-a747-c11c684a2ceb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5d059ac-6de0-4bac-a747-c11c684a2ceb-plugins-config-volume\") pod \"c5d059ac-6de0-4bac-a747-c11c684a2ceb\" (UID: \"c5d059ac-6de0-4bac-a747-c11c684a2ceb\") " Oct 30 19:10:40.955336 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:40.955288 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5d059ac-6de0-4bac-a747-c11c684a2ceb-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c5d059ac-6de0-4bac-a747-c11c684a2ceb-routing-overlay-volume\") pod \"c5d059ac-6de0-4bac-a747-c11c684a2ceb\" (UID: \"c5d059ac-6de0-4bac-a747-c11c684a2ceb\") " Oct 30 19:10:40.956232 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:40.956021 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5d059ac-6de0-4bac-a747-c11c684a2ceb-routing-overlay-volume" pod "c5d059ac-6de0-4bac-a747-c11c684a2ceb" (UID: "c5d059ac-6de0-4bac-a747-c11c684a2ceb"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:10:40.956375 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:40.956298 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5d059ac-6de0-4bac-a747-c11c684a2ceb-plugins-config-volume" pod "c5d059ac-6de0-4bac-a747-c11c684a2ceb" (UID: "c5d059ac-6de0-4bac-a747-c11c684a2ceb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:10:40.956686 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:40.956657 6960 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5d059ac-6de0-4bac-a747-c11c684a2ceb-plugins-config-volume\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:10:40.956828 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:40.956691 6960 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c5d059ac-6de0-4bac-a747-c11c684a2ceb-routing-overlay-volume\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:10:40.964201 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:40.963835 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5d059ac-6de0-4bac-a747-c11c684a2ceb-kube-api-access-stdcq" pod "c5d059ac-6de0-4bac-a747-c11c684a2ceb" (UID: "c5d059ac-6de0-4bac-a747-c11c684a2ceb"). InnerVolumeSpecName "kube-api-access-stdcq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:10:40.964201 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:40.964122 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5d059ac-6de0-4bac-a747-c11c684a2ceb-provider-credentials" pod "c5d059ac-6de0-4bac-a747-c11c684a2ceb" (UID: "c5d059ac-6de0-4bac-a747-c11c684a2ceb"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:10:41.032383 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:41.032348 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:41.058095 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:41.058057 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stdcq\" (UniqueName: \"kubernetes.io/projected/c5d059ac-6de0-4bac-a747-c11c684a2ceb-kube-api-access-stdcq\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:10:41.058095 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:41.058093 6960 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c5d059ac-6de0-4bac-a747-c11c684a2ceb-provider-credentials\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:10:41.159649 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:41.159549 6960 generic.go:356] "Generic (PLEG): container finished" podID="c5d059ac-6de0-4bac-a747-c11c684a2ceb" containerID="bb6594b0d3780a870f441df18da437752a0299f0bf6bb2e1bcb7f943bc76c135" exitCode=0 Oct 30 19:10:41.159649 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:41.159627 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-6cdfmd5lg" event={"ID":"c5d059ac-6de0-4bac-a747-c11c684a2ceb","Type":"ContainerDied","Data":"bb6594b0d3780a870f441df18da437752a0299f0bf6bb2e1bcb7f943bc76c135"} Oct 30 19:10:41.160244 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:41.159716 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-6cdfmd5lg" event={"ID":"c5d059ac-6de0-4bac-a747-c11c684a2ceb","Type":"ContainerDied","Data":"363c6ae54ceefa332601255f84228130c1eb7b7dcdecc19bf7d3c4fd252a2571"} Oct 30 19:10:41.160244 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:41.159747 6960 scope.go:122] "RemoveContainer" containerID="bb6594b0d3780a870f441df18da437752a0299f0bf6bb2e1bcb7f943bc76c135" Oct 30 19:10:41.160244 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:41.160203 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-6cdfmd5lg" Oct 30 19:10:41.177505 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:41.177483 6960 scope.go:122] "RemoveContainer" containerID="bb6594b0d3780a870f441df18da437752a0299f0bf6bb2e1bcb7f943bc76c135" Oct 30 19:10:41.177811 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:10:41.177788 6960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb6594b0d3780a870f441df18da437752a0299f0bf6bb2e1bcb7f943bc76c135\": container with ID starting with bb6594b0d3780a870f441df18da437752a0299f0bf6bb2e1bcb7f943bc76c135 not found: ID does not exist" containerID="bb6594b0d3780a870f441df18da437752a0299f0bf6bb2e1bcb7f943bc76c135" Oct 30 19:10:41.177903 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:41.177821 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb6594b0d3780a870f441df18da437752a0299f0bf6bb2e1bcb7f943bc76c135"} err="failed to get container status \"bb6594b0d3780a870f441df18da437752a0299f0bf6bb2e1bcb7f943bc76c135\": rpc error: code = NotFound desc = could not find container \"bb6594b0d3780a870f441df18da437752a0299f0bf6bb2e1bcb7f943bc76c135\": container with ID starting with bb6594b0d3780a870f441df18da437752a0299f0bf6bb2e1bcb7f943bc76c135 not found: ID does not exist" Oct 30 19:10:41.188446 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:41.188417 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:41.197901 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:41.197868 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-6cdfmd5lg"] Oct 30 19:10:41.201557 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:41.201534 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-f7c6e3/payload-processing-external-model-e2e-extmod-b-f7c6e3-6cdfmd5lg"] Oct 30 19:10:41.397322 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:41.395627 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:41.512313 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:41.511852 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:42.193653 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:42.193610 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:42.310435 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:42.310399 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:42.530384 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:42.530301 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:42.631771 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:42.631739 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5d059ac-6de0-4bac-a747-c11c684a2ceb" path="/var/lib/kubelet/pods/c5d059ac-6de0-4bac-a747-c11c684a2ceb/volumes" Oct 30 19:10:42.870748 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:42.870717 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:43.187250 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:43.183755 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:43.574290 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:43.574260 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:43.722433 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:43.722398 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:43.840333 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:43.840250 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:44.452003 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:44.451970 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:45.682265 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:45.682194 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:46.186008 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:46.185972 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:46.300110 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:46.300066 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:46.717043 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:46.717008 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:47.613270 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:47.613235 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:48.424537 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:48.424502 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:49.031043 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:49.031008 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:49.159216 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:49.159181 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:49.606792 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:49.606753 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:50.655983 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:50.655934 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:50.810190 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:50.810135 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:51.461296 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:51.461261 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:51.645182 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:51.645115 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:51.830169 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:51.825688 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:51.996409 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:51.996362 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:52.330588 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:52.330399 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:52.485121 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:52.485085 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:53.042982 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:53.042944 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:53.166557 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:53.166450 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:53.684335 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:53.684303 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:53.824795 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:53.824764 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:53.951370 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:10:53.951333 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 19:10:53.973724 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:53.973684 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:54.141260 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:54.140881 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:54.701902 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:54.701873 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-685b2hwwf"] Oct 30 19:10:54.702686 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:54.702672 6960 cpu_manager.go:414] "containerMap: removing container" podUID="c5d059ac-6de0-4bac-a747-c11c684a2ceb" containerName="payload-processing" Oct 30 19:10:54.702686 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:54.702687 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5d059ac-6de0-4bac-a747-c11c684a2ceb" containerName="payload-processing" Oct 30 19:10:54.702803 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:54.702794 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5d059ac-6de0-4bac-a747-c11c684a2ceb" containerName="payload-processing" Oct 30 19:10:54.707772 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:54.707753 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-685b2hwwf" Oct 30 19:10:54.714293 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:54.714269 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a717c7\"/\"payload-processing-external-model-plugins-e2e-extmod-a-a717c7\"" Oct 30 19:10:54.714482 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:54.714433 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a717c7\"/\"kube-root-ca.crt\"" Oct 30 19:10:54.714574 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:54.714511 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a717c7\"/\"openshift-service-ca.crt\"" Oct 30 19:10:54.715418 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:54.715365 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a717c7\"/\"routing-overlay\"" Oct 30 19:10:54.719593 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:54.719568 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-685b2hwwf"] Oct 30 19:10:54.724079 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:54.724045 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a717c7\"/\"agc-cred-56b60f463986\"" Oct 30 19:10:54.724243 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:54.724203 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a717c7\"/\"agc-cred-1d8f79ff1400\"" Oct 30 19:10:54.724299 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:54.724254 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a717c7\"/\"payload-processing-external-model-e2e--ebe4ecff-dockercfg-vjqtd\"" Oct 30 19:10:54.787811 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:54.787781 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cb9d37f1-399e-41a0-8949-f360c3e64f13-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a717c7-685b2hwwf\" (UID: \"cb9d37f1-399e-41a0-8949-f360c3e64f13\") " pod="ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-685b2hwwf" Oct 30 19:10:54.787967 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:54.787870 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb9d37f1-399e-41a0-8949-f360c3e64f13-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a717c7-685b2hwwf\" (UID: \"cb9d37f1-399e-41a0-8949-f360c3e64f13\") " pod="ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-685b2hwwf" Oct 30 19:10:54.787967 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:54.787957 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8sqr\" (UniqueName: \"kubernetes.io/projected/cb9d37f1-399e-41a0-8949-f360c3e64f13-kube-api-access-k8sqr\") pod \"payload-processing-external-model-e2e-extmod-a-a717c7-685b2hwwf\" (UID: \"cb9d37f1-399e-41a0-8949-f360c3e64f13\") " pod="ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-685b2hwwf" Oct 30 19:10:54.788049 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:54.788010 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cb9d37f1-399e-41a0-8949-f360c3e64f13-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a717c7-685b2hwwf\" (UID: \"cb9d37f1-399e-41a0-8949-f360c3e64f13\") " pod="ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-685b2hwwf" Oct 30 19:10:54.819683 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:54.819656 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:54.892401 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:54.892356 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cb9d37f1-399e-41a0-8949-f360c3e64f13-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a717c7-685b2hwwf\" (UID: \"cb9d37f1-399e-41a0-8949-f360c3e64f13\") " pod="ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-685b2hwwf" Oct 30 19:10:54.892640 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:54.892469 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb9d37f1-399e-41a0-8949-f360c3e64f13-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a717c7-685b2hwwf\" (UID: \"cb9d37f1-399e-41a0-8949-f360c3e64f13\") " pod="ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-685b2hwwf" Oct 30 19:10:54.892640 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:54.892564 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8sqr\" (UniqueName: \"kubernetes.io/projected/cb9d37f1-399e-41a0-8949-f360c3e64f13-kube-api-access-k8sqr\") pod \"payload-processing-external-model-e2e-extmod-a-a717c7-685b2hwwf\" (UID: \"cb9d37f1-399e-41a0-8949-f360c3e64f13\") " pod="ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-685b2hwwf" Oct 30 19:10:54.892640 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:54.892628 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cb9d37f1-399e-41a0-8949-f360c3e64f13-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a717c7-685b2hwwf\" (UID: \"cb9d37f1-399e-41a0-8949-f360c3e64f13\") " pod="ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-685b2hwwf" Oct 30 19:10:54.893650 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:54.893583 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cb9d37f1-399e-41a0-8949-f360c3e64f13-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a717c7-685b2hwwf\" (UID: \"cb9d37f1-399e-41a0-8949-f360c3e64f13\") " pod="ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-685b2hwwf" Oct 30 19:10:54.895016 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:54.894977 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb9d37f1-399e-41a0-8949-f360c3e64f13-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a717c7-685b2hwwf\" (UID: \"cb9d37f1-399e-41a0-8949-f360c3e64f13\") " pod="ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-685b2hwwf" Oct 30 19:10:54.899944 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:54.899922 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cb9d37f1-399e-41a0-8949-f360c3e64f13-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a717c7-685b2hwwf\" (UID: \"cb9d37f1-399e-41a0-8949-f360c3e64f13\") " pod="ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-685b2hwwf" Oct 30 19:10:54.934248 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:54.934204 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8sqr\" (UniqueName: \"kubernetes.io/projected/cb9d37f1-399e-41a0-8949-f360c3e64f13-kube-api-access-k8sqr\") pod \"payload-processing-external-model-e2e-extmod-a-a717c7-685b2hwwf\" (UID: \"cb9d37f1-399e-41a0-8949-f360c3e64f13\") " pod="ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-685b2hwwf" Oct 30 19:10:55.027709 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:55.027638 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-685b2hwwf" Oct 30 19:10:55.345160 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:55.344363 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-685b2hwwf"] Oct 30 19:10:55.390909 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:55.390876 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-685b2hwwf" event={"ID":"cb9d37f1-399e-41a0-8949-f360c3e64f13","Type":"ContainerStarted","Data":"625a8c3722d3208b852424a76f4ae95053ddffc5d601334533997f86a8f08b88"} Oct 30 19:10:55.834557 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:55.834516 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:56.399725 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:56.399682 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-685b2hwwf" event={"ID":"cb9d37f1-399e-41a0-8949-f360c3e64f13","Type":"ContainerStarted","Data":"4f3eabe5aff5be547de096043468a53a4b726ce3666df4712d481b37a26ce191"} Oct 30 19:10:56.400268 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:56.399913 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-685b2hwwf" Oct 30 19:10:56.420488 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:56.420423 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-685b2hwwf" podStartSLOduration=2.420410165 podStartE2EDuration="2.420410165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:10:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:10:56.417988933 +0000 UTC m=+2223.947020072" watchObservedRunningTime="2026-10-30 19:10:56.420410165 +0000 UTC m=+2223.949441304" Oct 30 19:10:56.892575 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:56.892534 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:57.550031 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:57.549988 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:57.712632 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:57.712599 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:57.835205 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:57.835172 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:58.634682 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:58.634640 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:10:59.416062 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:10:59.416024 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:00.302172 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:00.302125 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:01.240251 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:01.240208 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:02.089950 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:02.089897 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:02.899803 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:02.899773 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:03.246053 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:03.245960 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:03.359772 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:03.359718 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:04.369870 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:04.369833 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:05.039260 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:05.039217 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:05.211658 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:05.211622 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:06.043026 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:06.042991 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:06.923073 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:06.923040 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:07.410983 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:07.410954 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-685b2hwwf" Oct 30 19:11:07.705004 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:07.704923 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:08.624976 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:08.624936 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:09.528423 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:09.528381 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:09.929907 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:09.929874 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:10.071104 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:10.071029 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:10.863961 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:10.863924 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:11.503660 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:11.503612 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:12.405360 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:12.405325 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:12.878118 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:12.878062 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:13.813322 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:13.813288 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:14.890228 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:14.890175 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:16.092464 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:16.092417 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:17.019925 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:17.019871 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:17.968334 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:17.963531 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:18.806479 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:18.806429 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:19.728828 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:19.728779 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:20.598250 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:20.598218 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:21.507489 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:21.507448 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:21.877987 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:21.877947 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:21.986563 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:21.986490 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:22.571614 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:22.571565 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:22.718242 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:22.718210 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:23.624527 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:23.624494 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:23.934666 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:11:23.934584 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 19:11:24.565054 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:24.565015 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:25.514112 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:25.514082 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:26.319434 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:26.319393 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:27.115707 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:27.115673 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:27.921283 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:27.921245 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:28.878421 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:28.878371 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:29.639688 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:29.639656 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:29.759454 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:29.759400 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:29.880686 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:29.880650 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:30.015919 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:30.015826 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:30.274684 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:30.274589 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:30.539922 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:30.539840 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:30.715948 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:30.715914 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:30.881264 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:30.881219 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:31.050580 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:31.050541 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:31.166419 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:31.166325 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:31.387774 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:31.387741 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:31.535007 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:31.534923 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:32.106354 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:32.106325 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:32.268165 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:32.266338 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:32.878672 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:32.878638 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:33.007769 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:33.007734 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:33.119718 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:33.119682 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:34.109538 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:34.109437 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:35.007478 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:35.007446 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:35.698203 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:35.698129 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:35.839674 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:35.839642 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:36.174597 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:36.174563 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:36.710898 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:36.710812 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:36.862737 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:36.862702 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:37.186440 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:37.186407 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:37.325359 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:37.325327 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:37.456423 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:37.456331 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:37.669793 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:37.669762 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:37.810787 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:37.810752 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:38.483045 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:38.483002 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:38.695173 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:38.691185 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:39.276979 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:39.276938 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:39.529888 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:39.529800 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:40.402100 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:40.402057 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:40.975818 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:40.975784 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:41.249066 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:41.248967 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:41.368974 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:41.368905 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:41.622864 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:41.621001 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:41.953736 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:41.953631 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:43.122105 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:43.122063 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:43.986258 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:43.986200 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:44.854216 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:44.854183 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:45.723423 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:45.723392 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:46.609558 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:46.609517 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:47.607834 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:47.607790 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:47.719646 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:47.719610 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:48.978943 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:48.978905 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:49.807717 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:49.807686 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:50.622829 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:50.622793 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:51.419294 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:51.419258 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:52.313919 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:52.313888 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:53.220760 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:53.220721 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:53.775565 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:53.775498 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:53.937454 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:11:53.937416 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 19:11:54.199697 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:54.199659 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:55.035888 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:55.035854 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:55.905375 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:55.905333 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:56.724528 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:56.724493 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:57.609053 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:57.609015 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:58.466724 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:58.466684 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:59.144737 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:59.144699 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:59.341685 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:59.341649 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:59.573813 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:59.573781 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:11:59.683465 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:11:59.683426 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:00.200588 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:12:00.200556 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:00.558000 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:12:00.557970 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:01.523878 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:12:01.523837 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:02.291163 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:12:02.291119 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:03.155281 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:12:03.155238 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:03.941855 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:12:03.941824 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:04.877404 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:12:04.877357 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:05.847909 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:12:05.847862 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:06.822845 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:12:06.822803 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:07.346282 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:12:07.346247 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:07.449110 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:12:07.449068 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:07.579792 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:12:07.579759 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:07.698230 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:12:07.696090 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:08.648824 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:12:08.648788 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:09.395880 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:12:09.395840 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:10.299334 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:12:10.299297 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:11.108942 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:12:11.108906 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:11.904519 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:12:11.904486 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:12.720587 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:12:12.720551 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:13.607806 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:12:13.607771 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:14.731192 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:12:14.720088 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:14.976921 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:12:14.976890 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:15.114129 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:12:15.114095 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:15.250489 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:12:15.250455 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:15.558229 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:12:15.555537 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:15.707771 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:12:15.707741 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:15.824216 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:12:15.821569 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:15.967035 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:12:15.967002 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:16.101498 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:12:16.101413 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:16.825221 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:12:16.825135 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:16.941793 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:12:16.941760 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:17.579930 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:12:17.579894 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:17.914179 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:12:17.914082 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:19.084434 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:12:19.084387 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:19.942066 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:12:19.942022 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:20.053030 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:12:20.052995 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:20.187228 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:12:20.182807 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:21.384417 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:12:21.384383 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:22.259512 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:12:22.259477 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:23.132255 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:12:23.132220 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:23.939963 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:12:23.939931 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 19:12:23.996085 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:12:23.996046 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:24.815756 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:12:24.815728 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:25.598388 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:12:25.598349 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:26.561268 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:12:26.561232 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:27.365771 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:12:27.365737 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:28.202796 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:12:28.202748 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:29.109448 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:12:29.109356 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:29.895276 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:12:29.895240 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:30.850623 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:12:30.850528 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:31.718425 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:12:31.718382 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:32.525563 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:12:32.525524 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:33.432984 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:12:33.432937 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:34.219530 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:12:34.219492 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:35.240627 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:12:35.240588 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:36.019953 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:12:36.019914 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:36.838276 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:12:36.838229 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:37.717825 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:12:37.717789 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:38.599388 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:12:38.599357 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:39.402203 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:12:39.402172 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:40.214996 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:12:40.214964 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:41.284419 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:12:41.284373 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:42.095872 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:12:42.095834 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:42.983786 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:12:42.983739 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:43.816513 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:12:43.816482 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:44.613009 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:12:44.612973 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:45.429710 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:12:45.429674 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:46.205744 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:12:46.205706 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:47.180721 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:12:47.180660 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:48.311817 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:12:48.311781 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:49.260569 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:12:49.260536 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:50.133938 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:12:50.133907 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:51.078283 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:12:51.078247 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:52.102044 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:12:52.102010 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:53.020831 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:12:53.020794 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:53.830201 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:12:53.830133 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:53.947316 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:12:53.947278 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:53.965076 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:12:53.965003 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 19:12:54.978762 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:12:54.978733 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:55.742125 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:12:55.742089 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:56.521313 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:12:56.521279 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:57.119842 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:12:57.119807 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:57.419015 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:12:57.418933 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:58.216088 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:12:58.216050 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:59.018958 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:12:59.018916 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:12:59.934205 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:12:59.934138 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:00.812270 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:00.812239 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:01.718608 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:01.718575 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:02.623393 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:02.623359 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:03.651556 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:03.651461 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:04.194593 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:04.194556 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:04.359168 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:04.358711 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:05.058323 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:05.058258 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:05.871816 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:05.871786 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:06.740844 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:06.740796 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:07.497317 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:07.497280 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:08.374614 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:08.374576 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:09.390356 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:09.390312 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:10.179917 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:10.179880 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:11.335506 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:11.335478 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:12.324305 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:12.324235 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:12.741771 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:12.741678 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:12.857777 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:12.857705 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:13.492065 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:13.492019 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:13.835755 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:13.835726 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-685b2hwwf"] Oct 30 19:13:13.869888 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:13.869855 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:14.011261 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:14.011214 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:14.489217 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:14.489188 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:14.688914 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:14.688876 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:15.531902 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:15.531874 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:16.411022 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:16.410991 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:16.919473 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:16.919438 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:17.060292 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:17.060255 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:17.554529 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:17.554477 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:17.780660 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:17.780627 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:18.741094 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:18.741054 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:18.837378 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:18.837332 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-685b2hwwf" podUID="cb9d37f1-399e-41a0-8949-f360c3e64f13" containerName="payload-processing" containerID="cri-o://4f3eabe5aff5be547de096043468a53a4b726ce3666df4712d481b37a26ce191" gracePeriod=25 Oct 30 19:13:19.140607 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:19.140579 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-685b2hwwf" Oct 30 19:13:19.198947 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:19.198906 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb9d37f1-399e-41a0-8949-f360c3e64f13-kube-api-access-k8sqr\" (UniqueName: \"kubernetes.io/projected/cb9d37f1-399e-41a0-8949-f360c3e64f13-kube-api-access-k8sqr\") pod \"cb9d37f1-399e-41a0-8949-f360c3e64f13\" (UID: \"cb9d37f1-399e-41a0-8949-f360c3e64f13\") " Oct 30 19:13:19.199212 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:19.199038 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb9d37f1-399e-41a0-8949-f360c3e64f13-provider-credentials\" (UniqueName: \"kubernetes.io/projected/cb9d37f1-399e-41a0-8949-f360c3e64f13-provider-credentials\") pod \"cb9d37f1-399e-41a0-8949-f360c3e64f13\" (UID: \"cb9d37f1-399e-41a0-8949-f360c3e64f13\") " Oct 30 19:13:19.199212 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:19.199078 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb9d37f1-399e-41a0-8949-f360c3e64f13-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb9d37f1-399e-41a0-8949-f360c3e64f13-plugins-config-volume\") pod \"cb9d37f1-399e-41a0-8949-f360c3e64f13\" (UID: \"cb9d37f1-399e-41a0-8949-f360c3e64f13\") " Oct 30 19:13:19.199428 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:19.199402 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb9d37f1-399e-41a0-8949-f360c3e64f13-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cb9d37f1-399e-41a0-8949-f360c3e64f13-routing-overlay-volume\") pod \"cb9d37f1-399e-41a0-8949-f360c3e64f13\" (UID: \"cb9d37f1-399e-41a0-8949-f360c3e64f13\") " Oct 30 19:13:19.199574 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:19.199544 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb9d37f1-399e-41a0-8949-f360c3e64f13-plugins-config-volume" pod "cb9d37f1-399e-41a0-8949-f360c3e64f13" (UID: "cb9d37f1-399e-41a0-8949-f360c3e64f13"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:13:19.199823 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:19.199800 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb9d37f1-399e-41a0-8949-f360c3e64f13-routing-overlay-volume" pod "cb9d37f1-399e-41a0-8949-f360c3e64f13" (UID: "cb9d37f1-399e-41a0-8949-f360c3e64f13"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:13:19.200383 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:19.200361 6960 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cb9d37f1-399e-41a0-8949-f360c3e64f13-routing-overlay-volume\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:13:19.200470 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:19.200390 6960 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb9d37f1-399e-41a0-8949-f360c3e64f13-plugins-config-volume\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:13:19.201598 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:19.201570 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb9d37f1-399e-41a0-8949-f360c3e64f13-kube-api-access-k8sqr" pod "cb9d37f1-399e-41a0-8949-f360c3e64f13" (UID: "cb9d37f1-399e-41a0-8949-f360c3e64f13"). InnerVolumeSpecName "kube-api-access-k8sqr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:13:19.201716 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:19.201683 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb9d37f1-399e-41a0-8949-f360c3e64f13-provider-credentials" pod "cb9d37f1-399e-41a0-8949-f360c3e64f13" (UID: "cb9d37f1-399e-41a0-8949-f360c3e64f13"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:13:19.302138 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:19.302044 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8sqr\" (UniqueName: \"kubernetes.io/projected/cb9d37f1-399e-41a0-8949-f360c3e64f13-kube-api-access-k8sqr\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:13:19.302138 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:19.302079 6960 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cb9d37f1-399e-41a0-8949-f360c3e64f13-provider-credentials\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:13:19.688290 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:19.687767 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:19.868618 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:19.868586 6960 generic.go:356] "Generic (PLEG): container finished" podID="cb9d37f1-399e-41a0-8949-f360c3e64f13" containerID="4f3eabe5aff5be547de096043468a53a4b726ce3666df4712d481b37a26ce191" exitCode=0 Oct 30 19:13:19.869113 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:19.868643 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-685b2hwwf" Oct 30 19:13:19.869113 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:19.868663 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-685b2hwwf" event={"ID":"cb9d37f1-399e-41a0-8949-f360c3e64f13","Type":"ContainerDied","Data":"4f3eabe5aff5be547de096043468a53a4b726ce3666df4712d481b37a26ce191"} Oct 30 19:13:19.869113 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:19.868732 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-685b2hwwf" event={"ID":"cb9d37f1-399e-41a0-8949-f360c3e64f13","Type":"ContainerDied","Data":"625a8c3722d3208b852424a76f4ae95053ddffc5d601334533997f86a8f08b88"} Oct 30 19:13:19.869113 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:19.868766 6960 scope.go:122] "RemoveContainer" containerID="4f3eabe5aff5be547de096043468a53a4b726ce3666df4712d481b37a26ce191" Oct 30 19:13:19.881334 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:19.881317 6960 scope.go:122] "RemoveContainer" containerID="4f3eabe5aff5be547de096043468a53a4b726ce3666df4712d481b37a26ce191" Oct 30 19:13:19.881742 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:13:19.881716 6960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f3eabe5aff5be547de096043468a53a4b726ce3666df4712d481b37a26ce191\": container with ID starting with 4f3eabe5aff5be547de096043468a53a4b726ce3666df4712d481b37a26ce191 not found: ID does not exist" containerID="4f3eabe5aff5be547de096043468a53a4b726ce3666df4712d481b37a26ce191" Oct 30 19:13:19.881865 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:19.881754 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f3eabe5aff5be547de096043468a53a4b726ce3666df4712d481b37a26ce191"} err="failed to get container status \"4f3eabe5aff5be547de096043468a53a4b726ce3666df4712d481b37a26ce191\": rpc error: code = NotFound desc = could not find container \"4f3eabe5aff5be547de096043468a53a4b726ce3666df4712d481b37a26ce191\": container with ID starting with 4f3eabe5aff5be547de096043468a53a4b726ce3666df4712d481b37a26ce191 not found: ID does not exist" Oct 30 19:13:19.894385 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:19.894362 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-685b2hwwf"] Oct 30 19:13:19.899218 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:19.899195 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-a717c7/payload-processing-external-model-e2e-extmod-a-a717c7-685b2hwwf"] Oct 30 19:13:20.595175 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:20.595132 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb9d37f1-399e-41a0-8949-f360c3e64f13" path="/var/lib/kubelet/pods/cb9d37f1-399e-41a0-8949-f360c3e64f13/volumes" Oct 30 19:13:20.914384 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:20.914300 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:21.718582 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:21.718539 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:22.103409 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:22.103376 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:22.245865 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:22.245824 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:22.688875 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:22.688831 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:23.609922 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:23.609891 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:23.938969 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:13:23.938885 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 19:13:24.024613 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:24.024581 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:24.170246 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:24.170169 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:24.314176 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:24.313995 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:24.500592 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:24.500559 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:24.700460 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:24.700357 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:24.814705 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:24.814669 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:24.954361 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:24.954268 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:25.085379 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:25.085347 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:25.460561 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:25.460522 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:25.613248 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:25.613190 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:25.727299 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:25.727226 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:26.135429 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:26.135391 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:26.344696 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:26.344654 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:27.287405 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:27.287374 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:28.089366 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:28.089329 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:28.905686 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:28.905655 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:29.048632 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:29.048594 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:29.884763 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:29.884730 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:29.999699 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:29.999640 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:30.614746 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:30.614709 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:31.086684 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:31.086652 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:32.031854 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:32.031813 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:32.742064 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:32.742027 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:32.874383 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:32.874349 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:33.005507 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:33.005424 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:33.646631 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:33.646600 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:33.756832 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:33.756753 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:33.981950 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:33.981626 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:34.128119 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:34.128088 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:34.247644 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:34.247551 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:34.375740 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:34.375704 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:35.552371 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:35.552303 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:36.603777 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:36.603735 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:37.432210 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:37.432167 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:38.281456 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:38.281416 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:39.047779 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:39.047749 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:39.921602 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:39.921564 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:40.863867 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:40.863827 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:40.986243 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:40.986198 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:41.748329 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:41.748290 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:41.892998 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:41.892961 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:42.034796 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:42.034707 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:43.251837 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:43.251792 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:44.346595 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:44.346559 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:45.214241 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:45.214194 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:46.086606 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:46.086568 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:47.088172 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:47.088085 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:48.449621 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:48.449581 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:49.323035 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:49.322761 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:50.849496 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:50.849464 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:51.969833 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:51.969788 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:52.983300 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:52.983256 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:53.892363 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:53.892328 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:53.990459 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:13:53.990398 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 19:13:53.998026 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:53.997996 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:54.697842 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:54.697808 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:55.559090 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:55.559056 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:56.489392 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:56.489357 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:57.311724 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:57.311679 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:58.191717 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:58.191681 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:58.313725 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:58.313690 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-gtlvx"] Oct 30 19:13:58.314205 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:58.313927 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-gtlvx" podUID="2c4cb031-1d4b-4026-8846-bcf16e2b5972" containerName="istio-proxy" containerID="cri-o://90a64aed40ad340601508841f812164bbbc9a9e60ce1d5156f5bd51d3dfbe9c1" gracePeriod=30 Oct 30 19:13:58.994100 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:58.994063 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:13:59.792528 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:13:59.792463 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:00.389188 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:00.389138 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-m5wdn"] Oct 30 19:14:00.389364 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:00.389340 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-m5wdn" podUID="6178626a-4928-40ee-b26a-362f39ee1965" containerName="istio-proxy" containerID="cri-o://163b6435eaaf59fbcfac836f9281149af66925cf50176ad67369ad5140812263" gracePeriod=30 Oct 30 19:14:00.720501 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:00.720402 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:01.704641 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:01.704604 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:02.274209 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:02.274131 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:02.392729 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:02.392694 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:02.865732 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:02.865694 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:02.998677 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:02.998643 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:03.455271 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:03.455237 6960 generic.go:356] "Generic (PLEG): container finished" podID="2c4cb031-1d4b-4026-8846-bcf16e2b5972" containerID="90a64aed40ad340601508841f812164bbbc9a9e60ce1d5156f5bd51d3dfbe9c1" exitCode=0 Oct 30 19:14:03.455440 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:03.455309 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-gtlvx" event={"ID":"2c4cb031-1d4b-4026-8846-bcf16e2b5972","Type":"ContainerDied","Data":"90a64aed40ad340601508841f812164bbbc9a9e60ce1d5156f5bd51d3dfbe9c1"} Oct 30 19:14:03.614342 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:03.614313 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-gtlvx" Oct 30 19:14:03.788820 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:03.788787 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c4cb031-1d4b-4026-8846-bcf16e2b5972-kube-api-access-6jnx6\" (UniqueName: \"kubernetes.io/projected/2c4cb031-1d4b-4026-8846-bcf16e2b5972-kube-api-access-6jnx6\") pod \"2c4cb031-1d4b-4026-8846-bcf16e2b5972\" (UID: \"2c4cb031-1d4b-4026-8846-bcf16e2b5972\") " Oct 30 19:14:03.789030 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:03.788878 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c4cb031-1d4b-4026-8846-bcf16e2b5972-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c4cb031-1d4b-4026-8846-bcf16e2b5972-istiod-ca-cert\") pod \"2c4cb031-1d4b-4026-8846-bcf16e2b5972\" (UID: \"2c4cb031-1d4b-4026-8846-bcf16e2b5972\") " Oct 30 19:14:03.789030 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:03.788916 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c4cb031-1d4b-4026-8846-bcf16e2b5972-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c4cb031-1d4b-4026-8846-bcf16e2b5972-workload-socket\") pod \"2c4cb031-1d4b-4026-8846-bcf16e2b5972\" (UID: \"2c4cb031-1d4b-4026-8846-bcf16e2b5972\") " Oct 30 19:14:03.789030 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:03.788957 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c4cb031-1d4b-4026-8846-bcf16e2b5972-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c4cb031-1d4b-4026-8846-bcf16e2b5972-credential-socket\") pod \"2c4cb031-1d4b-4026-8846-bcf16e2b5972\" (UID: \"2c4cb031-1d4b-4026-8846-bcf16e2b5972\") " Oct 30 19:14:03.789030 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:03.789001 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c4cb031-1d4b-4026-8846-bcf16e2b5972-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c4cb031-1d4b-4026-8846-bcf16e2b5972-workload-certs\") pod \"2c4cb031-1d4b-4026-8846-bcf16e2b5972\" (UID: \"2c4cb031-1d4b-4026-8846-bcf16e2b5972\") " Oct 30 19:14:03.789331 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:03.789063 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c4cb031-1d4b-4026-8846-bcf16e2b5972-istio-token\" (UniqueName: \"kubernetes.io/projected/2c4cb031-1d4b-4026-8846-bcf16e2b5972-istio-token\") pod \"2c4cb031-1d4b-4026-8846-bcf16e2b5972\" (UID: \"2c4cb031-1d4b-4026-8846-bcf16e2b5972\") " Oct 30 19:14:03.789331 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:03.789290 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c4cb031-1d4b-4026-8846-bcf16e2b5972-workload-socket" pod "2c4cb031-1d4b-4026-8846-bcf16e2b5972" (UID: "2c4cb031-1d4b-4026-8846-bcf16e2b5972"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:14:03.789331 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:03.789318 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c4cb031-1d4b-4026-8846-bcf16e2b5972-credential-socket" pod "2c4cb031-1d4b-4026-8846-bcf16e2b5972" (UID: "2c4cb031-1d4b-4026-8846-bcf16e2b5972"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:14:03.789494 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:03.789473 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2c4cb031-1d4b-4026-8846-bcf16e2b5972-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c4cb031-1d4b-4026-8846-bcf16e2b5972-istio-podinfo\") pod \"2c4cb031-1d4b-4026-8846-bcf16e2b5972\" (UID: \"2c4cb031-1d4b-4026-8846-bcf16e2b5972\") " Oct 30 19:14:03.789568 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:03.789521 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c4cb031-1d4b-4026-8846-bcf16e2b5972-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c4cb031-1d4b-4026-8846-bcf16e2b5972-istio-envoy\") pod \"2c4cb031-1d4b-4026-8846-bcf16e2b5972\" (UID: \"2c4cb031-1d4b-4026-8846-bcf16e2b5972\") " Oct 30 19:14:03.789630 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:03.789593 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c4cb031-1d4b-4026-8846-bcf16e2b5972-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c4cb031-1d4b-4026-8846-bcf16e2b5972-istio-data\") pod \"2c4cb031-1d4b-4026-8846-bcf16e2b5972\" (UID: \"2c4cb031-1d4b-4026-8846-bcf16e2b5972\") " Oct 30 19:14:03.789630 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:03.789609 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c4cb031-1d4b-4026-8846-bcf16e2b5972-workload-certs" pod "2c4cb031-1d4b-4026-8846-bcf16e2b5972" (UID: "2c4cb031-1d4b-4026-8846-bcf16e2b5972"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:14:03.791116 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:03.790657 6960 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c4cb031-1d4b-4026-8846-bcf16e2b5972-workload-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:14:03.791116 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:03.790681 6960 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c4cb031-1d4b-4026-8846-bcf16e2b5972-credential-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:14:03.791116 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:03.790695 6960 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c4cb031-1d4b-4026-8846-bcf16e2b5972-workload-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:14:03.791116 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:03.790910 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c4cb031-1d4b-4026-8846-bcf16e2b5972-istio-data" pod "2c4cb031-1d4b-4026-8846-bcf16e2b5972" (UID: "2c4cb031-1d4b-4026-8846-bcf16e2b5972"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:14:03.792061 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:03.792032 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2c4cb031-1d4b-4026-8846-bcf16e2b5972-istio-podinfo" pod "2c4cb031-1d4b-4026-8846-bcf16e2b5972" (UID: "2c4cb031-1d4b-4026-8846-bcf16e2b5972"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:14:03.792181 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:03.792088 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c4cb031-1d4b-4026-8846-bcf16e2b5972-kube-api-access-6jnx6" pod "2c4cb031-1d4b-4026-8846-bcf16e2b5972" (UID: "2c4cb031-1d4b-4026-8846-bcf16e2b5972"). InnerVolumeSpecName "kube-api-access-6jnx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:14:03.792362 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:03.792330 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c4cb031-1d4b-4026-8846-bcf16e2b5972-istiod-ca-cert" pod "2c4cb031-1d4b-4026-8846-bcf16e2b5972" (UID: "2c4cb031-1d4b-4026-8846-bcf16e2b5972"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:14:03.793128 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:03.793108 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c4cb031-1d4b-4026-8846-bcf16e2b5972-istio-envoy" pod "2c4cb031-1d4b-4026-8846-bcf16e2b5972" (UID: "2c4cb031-1d4b-4026-8846-bcf16e2b5972"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:14:03.793316 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:03.793299 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c4cb031-1d4b-4026-8846-bcf16e2b5972-istio-token" pod "2c4cb031-1d4b-4026-8846-bcf16e2b5972" (UID: "2c4cb031-1d4b-4026-8846-bcf16e2b5972"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:14:03.892714 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:03.892627 6960 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c4cb031-1d4b-4026-8846-bcf16e2b5972-istio-podinfo\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:14:03.892714 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:03.892654 6960 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c4cb031-1d4b-4026-8846-bcf16e2b5972-istio-envoy\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:14:03.892714 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:03.892666 6960 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c4cb031-1d4b-4026-8846-bcf16e2b5972-istio-data\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:14:03.892714 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:03.892691 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jnx6\" (UniqueName: \"kubernetes.io/projected/2c4cb031-1d4b-4026-8846-bcf16e2b5972-kube-api-access-6jnx6\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:14:03.892714 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:03.892706 6960 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c4cb031-1d4b-4026-8846-bcf16e2b5972-istiod-ca-cert\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:14:03.892714 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:03.892714 6960 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c4cb031-1d4b-4026-8846-bcf16e2b5972-istio-token\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:14:03.898715 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:03.898687 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:04.463930 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:04.463901 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-gtlvx" Oct 30 19:14:04.464111 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:04.463894 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-gtlvx" event={"ID":"2c4cb031-1d4b-4026-8846-bcf16e2b5972","Type":"ContainerDied","Data":"f538e4b29462cb222a00369d4b7e3f38fd198dd45e8316c9f6cc4607983461a8"} Oct 30 19:14:04.464111 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:04.464044 6960 scope.go:122] "RemoveContainer" containerID="90a64aed40ad340601508841f812164bbbc9a9e60ce1d5156f5bd51d3dfbe9c1" Oct 30 19:14:04.492303 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:04.492275 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-gtlvx"] Oct 30 19:14:04.495780 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:04.495754 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-gtlvx"] Oct 30 19:14:04.580739 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:04.580707 6960 patch_prober.go:29] interesting pod/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-m5wdn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:14:04.580887 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:04.580768 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-m5wdn" podUID="6178626a-4928-40ee-b26a-362f39ee1965" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:14:04.595234 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:04.595210 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c4cb031-1d4b-4026-8846-bcf16e2b5972" path="/var/lib/kubelet/pods/2c4cb031-1d4b-4026-8846-bcf16e2b5972/volumes" Oct 30 19:14:04.699808 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:04.699773 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:05.468009 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:05.467977 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:05.723777 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:05.723755 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-m5wdn" Oct 30 19:14:05.824382 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:05.824354 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6178626a-4928-40ee-b26a-362f39ee1965-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6178626a-4928-40ee-b26a-362f39ee1965-istiod-ca-cert\") pod \"6178626a-4928-40ee-b26a-362f39ee1965\" (UID: \"6178626a-4928-40ee-b26a-362f39ee1965\") " Oct 30 19:14:05.824561 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:05.824390 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6178626a-4928-40ee-b26a-362f39ee1965-istio-token\" (UniqueName: \"kubernetes.io/projected/6178626a-4928-40ee-b26a-362f39ee1965-istio-token\") pod \"6178626a-4928-40ee-b26a-362f39ee1965\" (UID: \"6178626a-4928-40ee-b26a-362f39ee1965\") " Oct 30 19:14:05.824561 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:05.824419 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6178626a-4928-40ee-b26a-362f39ee1965-kube-api-access-nw4bw\" (UniqueName: \"kubernetes.io/projected/6178626a-4928-40ee-b26a-362f39ee1965-kube-api-access-nw4bw\") pod \"6178626a-4928-40ee-b26a-362f39ee1965\" (UID: \"6178626a-4928-40ee-b26a-362f39ee1965\") " Oct 30 19:14:05.824561 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:05.824556 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6178626a-4928-40ee-b26a-362f39ee1965-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6178626a-4928-40ee-b26a-362f39ee1965-workload-certs\") pod \"6178626a-4928-40ee-b26a-362f39ee1965\" (UID: \"6178626a-4928-40ee-b26a-362f39ee1965\") " Oct 30 19:14:05.824695 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:05.824648 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6178626a-4928-40ee-b26a-362f39ee1965-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6178626a-4928-40ee-b26a-362f39ee1965-istio-envoy\") pod \"6178626a-4928-40ee-b26a-362f39ee1965\" (UID: \"6178626a-4928-40ee-b26a-362f39ee1965\") " Oct 30 19:14:05.824775 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:05.824757 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6178626a-4928-40ee-b26a-362f39ee1965-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6178626a-4928-40ee-b26a-362f39ee1965-istio-data\") pod \"6178626a-4928-40ee-b26a-362f39ee1965\" (UID: \"6178626a-4928-40ee-b26a-362f39ee1965\") " Oct 30 19:14:05.824839 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:05.824805 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6178626a-4928-40ee-b26a-362f39ee1965-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6178626a-4928-40ee-b26a-362f39ee1965-istio-podinfo\") pod \"6178626a-4928-40ee-b26a-362f39ee1965\" (UID: \"6178626a-4928-40ee-b26a-362f39ee1965\") " Oct 30 19:14:05.824897 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:05.824869 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6178626a-4928-40ee-b26a-362f39ee1965-workload-certs" pod "6178626a-4928-40ee-b26a-362f39ee1965" (UID: "6178626a-4928-40ee-b26a-362f39ee1965"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:14:05.824960 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:05.824906 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6178626a-4928-40ee-b26a-362f39ee1965-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6178626a-4928-40ee-b26a-362f39ee1965-credential-socket\") pod \"6178626a-4928-40ee-b26a-362f39ee1965\" (UID: \"6178626a-4928-40ee-b26a-362f39ee1965\") " Oct 30 19:14:05.824960 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:05.824918 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6178626a-4928-40ee-b26a-362f39ee1965-istiod-ca-cert" pod "6178626a-4928-40ee-b26a-362f39ee1965" (UID: "6178626a-4928-40ee-b26a-362f39ee1965"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:14:05.825127 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:05.825100 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6178626a-4928-40ee-b26a-362f39ee1965-istio-data" pod "6178626a-4928-40ee-b26a-362f39ee1965" (UID: "6178626a-4928-40ee-b26a-362f39ee1965"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:14:05.825224 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:05.825178 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6178626a-4928-40ee-b26a-362f39ee1965-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6178626a-4928-40ee-b26a-362f39ee1965-workload-socket\") pod \"6178626a-4928-40ee-b26a-362f39ee1965\" (UID: \"6178626a-4928-40ee-b26a-362f39ee1965\") " Oct 30 19:14:05.825702 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:05.825656 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6178626a-4928-40ee-b26a-362f39ee1965-workload-socket" pod "6178626a-4928-40ee-b26a-362f39ee1965" (UID: "6178626a-4928-40ee-b26a-362f39ee1965"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:14:05.826334 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:05.826298 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6178626a-4928-40ee-b26a-362f39ee1965-credential-socket" pod "6178626a-4928-40ee-b26a-362f39ee1965" (UID: "6178626a-4928-40ee-b26a-362f39ee1965"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:14:05.826483 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:05.826462 6960 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6178626a-4928-40ee-b26a-362f39ee1965-credential-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:14:05.826531 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:05.826489 6960 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6178626a-4928-40ee-b26a-362f39ee1965-workload-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:14:05.826531 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:05.826505 6960 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6178626a-4928-40ee-b26a-362f39ee1965-istiod-ca-cert\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:14:05.826531 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:05.826520 6960 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6178626a-4928-40ee-b26a-362f39ee1965-workload-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:14:05.826637 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:05.826533 6960 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6178626a-4928-40ee-b26a-362f39ee1965-istio-data\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:14:05.827530 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:05.827506 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6178626a-4928-40ee-b26a-362f39ee1965-istio-podinfo" pod "6178626a-4928-40ee-b26a-362f39ee1965" (UID: "6178626a-4928-40ee-b26a-362f39ee1965"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:14:05.827649 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:05.827554 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6178626a-4928-40ee-b26a-362f39ee1965-istio-token" pod "6178626a-4928-40ee-b26a-362f39ee1965" (UID: "6178626a-4928-40ee-b26a-362f39ee1965"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:14:05.827712 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:05.827676 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6178626a-4928-40ee-b26a-362f39ee1965-istio-envoy" pod "6178626a-4928-40ee-b26a-362f39ee1965" (UID: "6178626a-4928-40ee-b26a-362f39ee1965"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:14:05.827784 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:05.827768 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6178626a-4928-40ee-b26a-362f39ee1965-kube-api-access-nw4bw" pod "6178626a-4928-40ee-b26a-362f39ee1965" (UID: "6178626a-4928-40ee-b26a-362f39ee1965"). InnerVolumeSpecName "kube-api-access-nw4bw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:14:05.927964 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:05.927934 6960 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6178626a-4928-40ee-b26a-362f39ee1965-istio-token\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:14:05.927964 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:05.927962 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nw4bw\" (UniqueName: \"kubernetes.io/projected/6178626a-4928-40ee-b26a-362f39ee1965-kube-api-access-nw4bw\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:14:05.928219 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:05.927973 6960 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6178626a-4928-40ee-b26a-362f39ee1965-istio-envoy\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:14:05.928219 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:05.927983 6960 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6178626a-4928-40ee-b26a-362f39ee1965-istio-podinfo\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:14:06.313954 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:06.313904 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:06.480862 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:06.480826 6960 generic.go:356] "Generic (PLEG): container finished" podID="6178626a-4928-40ee-b26a-362f39ee1965" containerID="163b6435eaaf59fbcfac836f9281149af66925cf50176ad67369ad5140812263" exitCode=0 Oct 30 19:14:06.481327 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:06.480896 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-m5wdn" Oct 30 19:14:06.481327 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:06.480904 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-m5wdn" event={"ID":"6178626a-4928-40ee-b26a-362f39ee1965","Type":"ContainerDied","Data":"163b6435eaaf59fbcfac836f9281149af66925cf50176ad67369ad5140812263"} Oct 30 19:14:06.481327 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:06.480996 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-m5wdn" event={"ID":"6178626a-4928-40ee-b26a-362f39ee1965","Type":"ContainerDied","Data":"89146352917a2fe0095a12dfc90950723a2a375d46b556ed6a8afeae57f749f4"} Oct 30 19:14:06.481327 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:06.481015 6960 scope.go:122] "RemoveContainer" containerID="163b6435eaaf59fbcfac836f9281149af66925cf50176ad67369ad5140812263" Oct 30 19:14:06.492502 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:06.492487 6960 scope.go:122] "RemoveContainer" containerID="163b6435eaaf59fbcfac836f9281149af66925cf50176ad67369ad5140812263" Oct 30 19:14:06.492842 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:14:06.492820 6960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"163b6435eaaf59fbcfac836f9281149af66925cf50176ad67369ad5140812263\": container with ID starting with 163b6435eaaf59fbcfac836f9281149af66925cf50176ad67369ad5140812263 not found: ID does not exist" containerID="163b6435eaaf59fbcfac836f9281149af66925cf50176ad67369ad5140812263" Oct 30 19:14:06.492935 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:06.492853 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"163b6435eaaf59fbcfac836f9281149af66925cf50176ad67369ad5140812263"} err="failed to get container status \"163b6435eaaf59fbcfac836f9281149af66925cf50176ad67369ad5140812263\": rpc error: code = NotFound desc = could not find container \"163b6435eaaf59fbcfac836f9281149af66925cf50176ad67369ad5140812263\": container with ID starting with 163b6435eaaf59fbcfac836f9281149af66925cf50176ad67369ad5140812263 not found: ID does not exist" Oct 30 19:14:06.505234 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:06.505213 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-m5wdn"] Oct 30 19:14:06.511074 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:06.511053 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-m5wdn"] Oct 30 19:14:06.595040 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:06.594968 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6178626a-4928-40ee-b26a-362f39ee1965" path="/var/lib/kubelet/pods/6178626a-4928-40ee-b26a-362f39ee1965/volumes" Oct 30 19:14:07.254713 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:07.254680 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:08.034754 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:08.034723 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:08.996521 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:08.996484 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:09.589582 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:09.589546 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:09.701604 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:09.701562 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:09.829783 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:09.829741 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:09.937846 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:09.937768 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:10.315510 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:10.315478 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zldp5"] Oct 30 19:14:10.316663 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:10.316637 6960 cpu_manager.go:414] "containerMap: removing container" podUID="2c4cb031-1d4b-4026-8846-bcf16e2b5972" containerName="istio-proxy" Oct 30 19:14:10.316663 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:10.316663 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c4cb031-1d4b-4026-8846-bcf16e2b5972" containerName="istio-proxy" Oct 30 19:14:10.316820 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:10.316700 6960 cpu_manager.go:414] "containerMap: removing container" podUID="cb9d37f1-399e-41a0-8949-f360c3e64f13" containerName="payload-processing" Oct 30 19:14:10.316820 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:10.316710 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb9d37f1-399e-41a0-8949-f360c3e64f13" containerName="payload-processing" Oct 30 19:14:10.316820 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:10.316765 6960 cpu_manager.go:414] "containerMap: removing container" podUID="6178626a-4928-40ee-b26a-362f39ee1965" containerName="istio-proxy" Oct 30 19:14:10.316820 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:10.316772 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6178626a-4928-40ee-b26a-362f39ee1965" containerName="istio-proxy" Oct 30 19:14:10.316990 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:10.316977 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c4cb031-1d4b-4026-8846-bcf16e2b5972" containerName="istio-proxy" Oct 30 19:14:10.317026 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:10.317000 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb9d37f1-399e-41a0-8949-f360c3e64f13" containerName="payload-processing" Oct 30 19:14:10.317026 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:10.317021 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="6178626a-4928-40ee-b26a-362f39ee1965" containerName="istio-proxy" Oct 30 19:14:10.323040 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:10.323016 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zldp5" Oct 30 19:14:10.367625 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:10.367578 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zldp5"] Oct 30 19:14:10.400114 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:10.400079 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d12464a9-d9c7-46e6-b3cc-ab0c16d75070-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-zldp5\" (UID: \"d12464a9-d9c7-46e6-b3cc-ab0c16d75070\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zldp5" Oct 30 19:14:10.400321 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:10.400138 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d12464a9-d9c7-46e6-b3cc-ab0c16d75070-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-zldp5\" (UID: \"d12464a9-d9c7-46e6-b3cc-ab0c16d75070\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zldp5" Oct 30 19:14:10.400454 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:10.400433 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d12464a9-d9c7-46e6-b3cc-ab0c16d75070-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-zldp5\" (UID: \"d12464a9-d9c7-46e6-b3cc-ab0c16d75070\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zldp5" Oct 30 19:14:10.400653 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:10.400627 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d12464a9-d9c7-46e6-b3cc-ab0c16d75070-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-zldp5\" (UID: \"d12464a9-d9c7-46e6-b3cc-ab0c16d75070\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zldp5" Oct 30 19:14:10.400718 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:10.400701 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d12464a9-d9c7-46e6-b3cc-ab0c16d75070-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-zldp5\" (UID: \"d12464a9-d9c7-46e6-b3cc-ab0c16d75070\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zldp5" Oct 30 19:14:10.400777 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:10.400761 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d12464a9-d9c7-46e6-b3cc-ab0c16d75070-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-zldp5\" (UID: \"d12464a9-d9c7-46e6-b3cc-ab0c16d75070\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zldp5" Oct 30 19:14:10.400853 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:10.400825 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d12464a9-d9c7-46e6-b3cc-ab0c16d75070-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-zldp5\" (UID: \"d12464a9-d9c7-46e6-b3cc-ab0c16d75070\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zldp5" Oct 30 19:14:10.400948 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:10.400933 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d12464a9-d9c7-46e6-b3cc-ab0c16d75070-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-zldp5\" (UID: \"d12464a9-d9c7-46e6-b3cc-ab0c16d75070\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zldp5" Oct 30 19:14:10.401007 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:10.400973 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95pld\" (UniqueName: \"kubernetes.io/projected/d12464a9-d9c7-46e6-b3cc-ab0c16d75070-kube-api-access-95pld\") pod \"maas-default-gateway-openshift-default-855f9574cf-zldp5\" (UID: \"d12464a9-d9c7-46e6-b3cc-ab0c16d75070\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zldp5" Oct 30 19:14:10.501944 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:10.501913 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d12464a9-d9c7-46e6-b3cc-ab0c16d75070-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-zldp5\" (UID: \"d12464a9-d9c7-46e6-b3cc-ab0c16d75070\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zldp5" Oct 30 19:14:10.502101 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:10.502029 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d12464a9-d9c7-46e6-b3cc-ab0c16d75070-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-zldp5\" (UID: \"d12464a9-d9c7-46e6-b3cc-ab0c16d75070\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zldp5" Oct 30 19:14:10.502101 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:10.502080 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d12464a9-d9c7-46e6-b3cc-ab0c16d75070-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-zldp5\" (UID: \"d12464a9-d9c7-46e6-b3cc-ab0c16d75070\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zldp5" Oct 30 19:14:10.502197 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:10.502119 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d12464a9-d9c7-46e6-b3cc-ab0c16d75070-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-zldp5\" (UID: \"d12464a9-d9c7-46e6-b3cc-ab0c16d75070\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zldp5" Oct 30 19:14:10.502197 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:10.502153 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d12464a9-d9c7-46e6-b3cc-ab0c16d75070-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-zldp5\" (UID: \"d12464a9-d9c7-46e6-b3cc-ab0c16d75070\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zldp5" Oct 30 19:14:10.502197 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:10.502185 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d12464a9-d9c7-46e6-b3cc-ab0c16d75070-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-zldp5\" (UID: \"d12464a9-d9c7-46e6-b3cc-ab0c16d75070\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zldp5" Oct 30 19:14:10.502326 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:10.502210 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95pld\" (UniqueName: \"kubernetes.io/projected/d12464a9-d9c7-46e6-b3cc-ab0c16d75070-kube-api-access-95pld\") pod \"maas-default-gateway-openshift-default-855f9574cf-zldp5\" (UID: \"d12464a9-d9c7-46e6-b3cc-ab0c16d75070\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zldp5" Oct 30 19:14:10.502326 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:10.502243 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d12464a9-d9c7-46e6-b3cc-ab0c16d75070-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-zldp5\" (UID: \"d12464a9-d9c7-46e6-b3cc-ab0c16d75070\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zldp5" Oct 30 19:14:10.502326 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:10.502278 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d12464a9-d9c7-46e6-b3cc-ab0c16d75070-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-zldp5\" (UID: \"d12464a9-d9c7-46e6-b3cc-ab0c16d75070\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zldp5" Oct 30 19:14:10.502431 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:10.502407 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d12464a9-d9c7-46e6-b3cc-ab0c16d75070-credential-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-zldp5\" (UID: \"d12464a9-d9c7-46e6-b3cc-ab0c16d75070\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zldp5" Oct 30 19:14:10.502482 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:10.502427 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d12464a9-d9c7-46e6-b3cc-ab0c16d75070-workload-socket\") pod \"maas-default-gateway-openshift-default-855f9574cf-zldp5\" (UID: \"d12464a9-d9c7-46e6-b3cc-ab0c16d75070\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zldp5" Oct 30 19:14:10.502572 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:10.502547 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d12464a9-d9c7-46e6-b3cc-ab0c16d75070-workload-certs\") pod \"maas-default-gateway-openshift-default-855f9574cf-zldp5\" (UID: \"d12464a9-d9c7-46e6-b3cc-ab0c16d75070\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zldp5" Oct 30 19:14:10.502821 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:10.502801 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d12464a9-d9c7-46e6-b3cc-ab0c16d75070-istio-data\") pod \"maas-default-gateway-openshift-default-855f9574cf-zldp5\" (UID: \"d12464a9-d9c7-46e6-b3cc-ab0c16d75070\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zldp5" Oct 30 19:14:10.503060 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:10.503043 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d12464a9-d9c7-46e6-b3cc-ab0c16d75070-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-855f9574cf-zldp5\" (UID: \"d12464a9-d9c7-46e6-b3cc-ab0c16d75070\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zldp5" Oct 30 19:14:10.504735 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:10.504715 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d12464a9-d9c7-46e6-b3cc-ab0c16d75070-istio-envoy\") pod \"maas-default-gateway-openshift-default-855f9574cf-zldp5\" (UID: \"d12464a9-d9c7-46e6-b3cc-ab0c16d75070\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zldp5" Oct 30 19:14:10.505104 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:10.505089 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d12464a9-d9c7-46e6-b3cc-ab0c16d75070-istio-podinfo\") pod \"maas-default-gateway-openshift-default-855f9574cf-zldp5\" (UID: \"d12464a9-d9c7-46e6-b3cc-ab0c16d75070\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zldp5" Oct 30 19:14:10.516016 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:10.515996 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d12464a9-d9c7-46e6-b3cc-ab0c16d75070-istio-token\") pod \"maas-default-gateway-openshift-default-855f9574cf-zldp5\" (UID: \"d12464a9-d9c7-46e6-b3cc-ab0c16d75070\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zldp5" Oct 30 19:14:10.517231 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:10.517213 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95pld\" (UniqueName: \"kubernetes.io/projected/d12464a9-d9c7-46e6-b3cc-ab0c16d75070-kube-api-access-95pld\") pod \"maas-default-gateway-openshift-default-855f9574cf-zldp5\" (UID: \"d12464a9-d9c7-46e6-b3cc-ab0c16d75070\") " pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zldp5" Oct 30 19:14:10.640073 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:10.639973 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zldp5" Oct 30 19:14:10.933072 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:10.933047 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zldp5"] Oct 30 19:14:10.933244 ip-10-0-140-163 kubenswrapper[6960]: W1030 19:14:10.933208 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd12464a9_d9c7_46e6_b3cc_ab0c16d75070.slice/crio-db5b1e2913586a856be066abfd6ffaf8b58942454fc7b23b9de4fd80b5439928 WatchSource:0}: Error finding container db5b1e2913586a856be066abfd6ffaf8b58942454fc7b23b9de4fd80b5439928: Status 404 returned error can't find the container with id db5b1e2913586a856be066abfd6ffaf8b58942454fc7b23b9de4fd80b5439928 Oct 30 19:14:10.936555 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:10.936507 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 19:14:10.936642 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:10.936592 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 19:14:10.936705 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:10.936663 6960 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 19:14:10.937030 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:10.936989 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:11.517850 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:11.517794 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:11.522711 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:11.522678 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zldp5" event={"ID":"d12464a9-d9c7-46e6-b3cc-ab0c16d75070","Type":"ContainerStarted","Data":"a1a996452ec360585f099b43945dfb0ba14d0909c89217ebd479c58f07816315"} Oct 30 19:14:11.522953 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:11.522936 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zldp5" event={"ID":"d12464a9-d9c7-46e6-b3cc-ab0c16d75070","Type":"ContainerStarted","Data":"db5b1e2913586a856be066abfd6ffaf8b58942454fc7b23b9de4fd80b5439928"} Oct 30 19:14:11.548579 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:11.548313 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zldp5" podStartSLOduration=1.548294067 podStartE2EDuration="1.548294067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:14:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:14:11.546817948 +0000 UTC m=+2419.075849089" watchObservedRunningTime="2026-10-30 19:14:11.548294067 +0000 UTC m=+2419.077325206" Oct 30 19:14:11.643170 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:11.641985 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zldp5" Oct 30 19:14:11.652055 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:11.652023 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:12.349826 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:12.349789 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:12.633601 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:12.633484 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:12.642772 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:12.642468 6960 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-zldp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 30 19:14:12.642772 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:12.642521 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zldp5" podUID="d12464a9-d9c7-46e6-b3cc-ab0c16d75070" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 30 19:14:12.783093 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:12.783052 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:13.219167 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:13.219042 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:13.640657 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:13.640629 6960 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-zldp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 30 19:14:13.640856 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:13.640696 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zldp5" podUID="d12464a9-d9c7-46e6-b3cc-ab0c16d75070" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 30 19:14:13.701666 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:13.701628 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:14.579772 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:14.579732 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:14.641413 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:14.641379 6960 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-zldp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 30 19:14:14.641711 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:14.641674 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zldp5" podUID="d12464a9-d9c7-46e6-b3cc-ab0c16d75070" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 30 19:14:15.426097 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:15.426057 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:15.640853 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:15.640818 6960 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-zldp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 30 19:14:15.641294 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:15.640877 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zldp5" podUID="d12464a9-d9c7-46e6-b3cc-ab0c16d75070" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 30 19:14:16.336594 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:16.336554 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:16.581993 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:16.581950 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:16.645116 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:16.640525 6960 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-zldp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 30 19:14:16.645792 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:16.645754 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zldp5" podUID="d12464a9-d9c7-46e6-b3cc-ab0c16d75070" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 30 19:14:16.847477 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:16.847444 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:17.640682 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:17.640647 6960 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-zldp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 30 19:14:17.640888 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:17.640712 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zldp5" podUID="d12464a9-d9c7-46e6-b3cc-ab0c16d75070" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 30 19:14:17.698543 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:17.694006 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:17.716617 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:17.716579 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-m4drm"] Oct 30 19:14:17.716855 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:17.716827 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-m4drm" podUID="e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684" containerName="istio-proxy" containerID="cri-o://faca51b8559e821ce29a2bd6f20555950e5384cd03d6e13aef06c772e400f514" gracePeriod=30 Oct 30 19:14:17.846825 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:17.845218 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:18.640684 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:18.640626 6960 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-zldp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 30 19:14:18.640923 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:18.640702 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zldp5" podUID="d12464a9-d9c7-46e6-b3cc-ab0c16d75070" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 30 19:14:19.150270 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:19.150231 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:19.640954 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:19.640921 6960 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-zldp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 30 19:14:19.641119 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:19.640986 6960 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zldp5" podUID="d12464a9-d9c7-46e6-b3cc-ab0c16d75070" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 30 19:14:20.221352 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:20.221319 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:20.328186 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:20.328115 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:20.521484 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:20.521401 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:20.700504 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:20.700474 6960 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zldp5" Oct 30 19:14:20.701035 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:20.701010 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zldp5" Oct 30 19:14:20.701890 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:20.701868 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zldp5" Oct 30 19:14:21.318438 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:21.318399 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:22.297397 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:22.297342 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:22.372500 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:22.372458 6960 patch_prober.go:29] interesting pod/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-m4drm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:14:22.372988 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:22.372557 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-m4drm" podUID="e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:14:23.018139 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:23.018110 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-m4drm" Oct 30 19:14:23.046797 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:23.046769 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684-credential-socket\") pod \"e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684\" (UID: \"e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684\") " Oct 30 19:14:23.046974 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:23.046952 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684-kube-api-access-47vjr\" (UniqueName: \"kubernetes.io/projected/e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684-kube-api-access-47vjr\") pod \"e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684\" (UID: \"e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684\") " Oct 30 19:14:23.047028 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:23.046996 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684-istio-envoy\") pod \"e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684\" (UID: \"e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684\") " Oct 30 19:14:23.047068 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:23.047037 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684-istio-token\" (UniqueName: \"kubernetes.io/projected/e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684-istio-token\") pod \"e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684\" (UID: \"e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684\") " Oct 30 19:14:23.047120 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:23.047060 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684-credential-socket" pod "e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684" (UID: "e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:14:23.047120 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:23.047071 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684-istiod-ca-cert\") pod \"e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684\" (UID: \"e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684\") " Oct 30 19:14:23.047407 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:23.047387 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684-istio-data\") pod \"e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684\" (UID: \"e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684\") " Oct 30 19:14:23.047495 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:23.047425 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684-workload-socket\") pod \"e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684\" (UID: \"e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684\") " Oct 30 19:14:23.047495 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:23.047454 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684-istiod-ca-cert" pod "e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684" (UID: "e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:14:23.047495 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:23.047459 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684-workload-certs\") pod \"e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684\" (UID: \"e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684\") " Oct 30 19:14:23.047639 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:23.047517 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684-istio-podinfo\") pod \"e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684\" (UID: \"e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684\") " Oct 30 19:14:23.047639 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:23.047589 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684-istio-data" pod "e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684" (UID: "e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:14:23.047953 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:23.047657 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684-workload-certs" pod "e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684" (UID: "e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:14:23.047953 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:23.047763 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684-workload-socket" pod "e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684" (UID: "e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:14:23.048356 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:23.048337 6960 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684-credential-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:14:23.048356 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:23.048357 6960 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684-istiod-ca-cert\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:14:23.048515 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:23.048367 6960 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684-istio-data\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:14:23.048515 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:23.048386 6960 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684-workload-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:14:23.048515 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:23.048398 6960 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684-workload-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:14:23.049691 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:23.049667 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684-kube-api-access-47vjr" pod "e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684" (UID: "e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684"). InnerVolumeSpecName "kube-api-access-47vjr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:14:23.050734 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:23.050307 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684-istio-token" pod "e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684" (UID: "e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:14:23.052500 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:23.052479 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684-istio-podinfo" pod "e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684" (UID: "e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:14:23.083351 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:23.082772 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684-istio-envoy" pod "e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684" (UID: "e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:14:23.149659 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:23.149629 6960 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684-istio-podinfo\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:14:23.149659 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:23.149655 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47vjr\" (UniqueName: \"kubernetes.io/projected/e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684-kube-api-access-47vjr\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:14:23.149840 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:23.149666 6960 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684-istio-envoy\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:14:23.149840 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:23.149677 6960 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684-istio-token\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:14:23.358749 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:23.358700 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:23.626531 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:23.626444 6960 generic.go:356] "Generic (PLEG): container finished" podID="e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684" containerID="faca51b8559e821ce29a2bd6f20555950e5384cd03d6e13aef06c772e400f514" exitCode=0 Oct 30 19:14:23.626531 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:23.626492 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-m4drm" Oct 30 19:14:23.627036 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:23.626532 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-m4drm" event={"ID":"e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684","Type":"ContainerDied","Data":"faca51b8559e821ce29a2bd6f20555950e5384cd03d6e13aef06c772e400f514"} Oct 30 19:14:23.627036 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:23.626589 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-m4drm" event={"ID":"e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684","Type":"ContainerDied","Data":"7cfba0a528c82321e6c1e149af0a8c80ff434aff0d0defbea042101638a4f0b6"} Oct 30 19:14:23.627036 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:23.626618 6960 scope.go:122] "RemoveContainer" containerID="faca51b8559e821ce29a2bd6f20555950e5384cd03d6e13aef06c772e400f514" Oct 30 19:14:23.639193 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:23.639171 6960 scope.go:122] "RemoveContainer" containerID="faca51b8559e821ce29a2bd6f20555950e5384cd03d6e13aef06c772e400f514" Oct 30 19:14:23.639451 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:14:23.639431 6960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"faca51b8559e821ce29a2bd6f20555950e5384cd03d6e13aef06c772e400f514\": container with ID starting with faca51b8559e821ce29a2bd6f20555950e5384cd03d6e13aef06c772e400f514 not found: ID does not exist" containerID="faca51b8559e821ce29a2bd6f20555950e5384cd03d6e13aef06c772e400f514" Oct 30 19:14:23.639517 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:23.639459 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"faca51b8559e821ce29a2bd6f20555950e5384cd03d6e13aef06c772e400f514"} err="failed to get container status \"faca51b8559e821ce29a2bd6f20555950e5384cd03d6e13aef06c772e400f514\": rpc error: code = NotFound desc = could not find container \"faca51b8559e821ce29a2bd6f20555950e5384cd03d6e13aef06c772e400f514\": container with ID starting with faca51b8559e821ce29a2bd6f20555950e5384cd03d6e13aef06c772e400f514 not found: ID does not exist" Oct 30 19:14:23.671743 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:23.671714 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-m4drm"] Oct 30 19:14:23.687482 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:23.687452 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-m4drm"] Oct 30 19:14:23.933355 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:14:23.933275 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 19:14:24.324265 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:24.324221 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:24.491168 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:24.490662 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:24.620241 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:24.617771 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684" path="/var/lib/kubelet/pods/e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684/volumes" Oct 30 19:14:25.162453 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:25.162415 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:26.226888 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:26.226822 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:27.420670 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:27.420639 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:27.584226 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:27.584189 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:27.719704 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:27.719605 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:27.962543 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:27.962512 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:28.295246 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:28.295120 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:28.501305 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:28.501273 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:28.644890 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:28.644856 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:28.814016 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:28.813865 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:28.938917 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:28.938758 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:29.140226 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:29.140123 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:29.771510 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:29.771468 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:30.297845 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:30.297789 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:30.406330 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:30.406296 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:30.705741 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:30.705709 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:31.100951 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:31.100785 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:32.298730 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:32.298691 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:32.851492 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:32.851455 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:33.487189 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:33.487062 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:33.595096 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:33.595060 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:34.628918 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:34.628882 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:35.373566 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:35.373529 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:35.500616 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:35.500580 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:35.791685 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:35.791603 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:36.839928 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:36.839877 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:37.638482 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:37.638439 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:38.423210 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:38.423178 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:39.243990 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:39.243941 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:40.098002 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:40.097965 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:40.836800 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:40.836752 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:41.654821 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:41.654768 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:42.597439 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:42.597401 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:43.400409 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:43.400374 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:44.447422 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:44.447381 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:45.209735 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:45.209704 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:46.022745 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:46.022712 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:46.931199 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:46.931123 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:47.812540 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:47.812510 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:48.611296 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:48.611255 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:49.399837 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:49.399801 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:50.461571 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:50.461539 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:51.518322 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:51.518295 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:52.484195 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:52.484122 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:53.534076 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:53.534044 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:53.934056 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:53.934022 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:53.936706 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:14:53.936670 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 19:14:54.417847 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:54.417816 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:55.203535 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:55.203502 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:55.956771 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:55.956717 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:56.826611 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:56.826565 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:57.702694 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:57.702661 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:58.564717 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:58.564685 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:14:59.396443 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:14:59.396403 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:00.324770 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:00.324695 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:00.454801 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:00.454768 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-c3a5b9-29889795-dz7tn"] Oct 30 19:15:00.456033 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:00.456009 6960 cpu_manager.go:414] "containerMap: removing container" podUID="e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684" containerName="istio-proxy" Oct 30 19:15:00.456132 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:00.456037 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684" containerName="istio-proxy" Oct 30 19:15:00.456303 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:00.456287 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9aeabaf-836f-4cfd-8f12-5fbe4e0dc684" containerName="istio-proxy" Oct 30 19:15:00.463013 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:00.462992 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-c3a5b9-29889795-dz7tn" Oct 30 19:15:00.478398 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:00.478365 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-75zkr\"" Oct 30 19:15:00.518977 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:00.518937 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-c3a5b9-29889795-dz7tn"] Oct 30 19:15:00.588992 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:00.588895 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c13c7df4-97b2-4a0d-855b-049561541df1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-c3a5b9-29889795-dz7tn\" (UID: \"c13c7df4-97b2-4a0d-855b-049561541df1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-c3a5b9-29889795-dz7tn" Oct 30 19:15:00.589217 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:00.588989 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8kqz\" (UniqueName: \"kubernetes.io/projected/c13c7df4-97b2-4a0d-855b-049561541df1-kube-api-access-k8kqz\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-c3a5b9-29889795-dz7tn\" (UID: \"c13c7df4-97b2-4a0d-855b-049561541df1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-c3a5b9-29889795-dz7tn" Oct 30 19:15:00.690786 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:00.690747 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c13c7df4-97b2-4a0d-855b-049561541df1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-c3a5b9-29889795-dz7tn\" (UID: \"c13c7df4-97b2-4a0d-855b-049561541df1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-c3a5b9-29889795-dz7tn" Oct 30 19:15:00.691055 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:00.690868 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8kqz\" (UniqueName: \"kubernetes.io/projected/c13c7df4-97b2-4a0d-855b-049561541df1-kube-api-access-k8kqz\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-c3a5b9-29889795-dz7tn\" (UID: \"c13c7df4-97b2-4a0d-855b-049561541df1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-c3a5b9-29889795-dz7tn" Oct 30 19:15:00.691367 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:00.691350 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c13c7df4-97b2-4a0d-855b-049561541df1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-c3a5b9-29889795-dz7tn\" (UID: \"c13c7df4-97b2-4a0d-855b-049561541df1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-c3a5b9-29889795-dz7tn" Oct 30 19:15:00.702670 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:00.702641 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8kqz\" (UniqueName: \"kubernetes.io/projected/c13c7df4-97b2-4a0d-855b-049561541df1-kube-api-access-k8kqz\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-c3a5b9-29889795-dz7tn\" (UID: \"c13c7df4-97b2-4a0d-855b-049561541df1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-c3a5b9-29889795-dz7tn" Oct 30 19:15:00.779307 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:00.779275 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-c3a5b9-29889795-dz7tn" Oct 30 19:15:01.065012 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:01.064982 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-c3a5b9-29889795-dz7tn"] Oct 30 19:15:01.065667 ip-10-0-140-163 kubenswrapper[6960]: W1030 19:15:01.065642 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc13c7df4_97b2_4a0d_855b_049561541df1.slice/crio-1cd45ebf93ac34b88c24a13cc4fe8726665bbf72ff610e47a52dfb45f610840f WatchSource:0}: Error finding container 1cd45ebf93ac34b88c24a13cc4fe8726665bbf72ff610e47a52dfb45f610840f: Status 404 returned error can't find the container with id 1cd45ebf93ac34b88c24a13cc4fe8726665bbf72ff610e47a52dfb45f610840f Oct 30 19:15:01.078941 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:01.078906 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-c3a5b9-29889795-dz7tn" event={"ID":"c13c7df4-97b2-4a0d-855b-049561541df1","Type":"ContainerStarted","Data":"1cd45ebf93ac34b88c24a13cc4fe8726665bbf72ff610e47a52dfb45f610840f"} Oct 30 19:15:01.506105 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:01.505898 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:02.088662 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:02.088624 6960 generic.go:356] "Generic (PLEG): container finished" podID="c13c7df4-97b2-4a0d-855b-049561541df1" containerID="a3e0fb9a6bc563a6b5b128e38266c1b89f2b096984ae28c2c02e70804dfe55f2" exitCode=60 Oct 30 19:15:02.088842 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:02.088689 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-c3a5b9-29889795-dz7tn" event={"ID":"c13c7df4-97b2-4a0d-855b-049561541df1","Type":"ContainerDied","Data":"a3e0fb9a6bc563a6b5b128e38266c1b89f2b096984ae28c2c02e70804dfe55f2"} Oct 30 19:15:02.089073 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:02.089058 6960 scope.go:122] "RemoveContainer" containerID="a3e0fb9a6bc563a6b5b128e38266c1b89f2b096984ae28c2c02e70804dfe55f2" Oct 30 19:15:02.332163 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:02.332118 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:03.100356 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:03.100312 6960 generic.go:356] "Generic (PLEG): container finished" podID="c13c7df4-97b2-4a0d-855b-049561541df1" containerID="5633f35b8513549165aac0b8c8a5205317c5683e49b7a619855342d68b0ee3cd" exitCode=60 Oct 30 19:15:03.100766 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:03.100455 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-c3a5b9-29889795-dz7tn" event={"ID":"c13c7df4-97b2-4a0d-855b-049561541df1","Type":"ContainerDied","Data":"5633f35b8513549165aac0b8c8a5205317c5683e49b7a619855342d68b0ee3cd"} Oct 30 19:15:03.100766 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:03.100532 6960 scope.go:122] "RemoveContainer" containerID="a3e0fb9a6bc563a6b5b128e38266c1b89f2b096984ae28c2c02e70804dfe55f2" Oct 30 19:15:03.100901 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:03.100870 6960 scope.go:122] "RemoveContainer" containerID="5633f35b8513549165aac0b8c8a5205317c5683e49b7a619855342d68b0ee3cd" Oct 30 19:15:03.101353 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:15:03.101323 6960 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-c3a5b9-29889795-dz7tn_odh-ai-gateway-infra(c13c7df4-97b2-4a0d-855b-049561541df1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-c3a5b9-29889795-dz7tn" podUID="c13c7df4-97b2-4a0d-855b-049561541df1" Oct 30 19:15:03.309224 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:03.309184 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:04.111325 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:04.111300 6960 scope.go:122] "RemoveContainer" containerID="5633f35b8513549165aac0b8c8a5205317c5683e49b7a619855342d68b0ee3cd" Oct 30 19:15:04.111712 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:15:04.111661 6960 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-c3a5b9-29889795-dz7tn_odh-ai-gateway-infra(c13c7df4-97b2-4a0d-855b-049561541df1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-c3a5b9-29889795-dz7tn" podUID="c13c7df4-97b2-4a0d-855b-049561541df1" Oct 30 19:15:04.270185 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:04.270122 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:05.168416 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:05.168381 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:06.232263 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:06.232221 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:07.156171 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:07.154780 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:07.909444 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:07.909408 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:09.042233 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:09.042134 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:09.906589 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:09.906552 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:10.729973 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:10.729940 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:11.556176 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:11.556125 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:12.471532 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:12.471490 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:13.286634 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:13.286599 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:14.133427 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:14.133386 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:15.092991 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:15.092954 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:15.586310 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:15.586284 6960 scope.go:122] "RemoveContainer" containerID="5633f35b8513549165aac0b8c8a5205317c5683e49b7a619855342d68b0ee3cd" Oct 30 19:15:15.991313 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:15.991283 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:16.229455 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:16.229353 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-c3a5b9-29889795-dz7tn" event={"ID":"c13c7df4-97b2-4a0d-855b-049561541df1","Type":"ContainerStarted","Data":"e458bf289428c91466fb3cd987cf7d1ceb1e4a2fb867f122273d79d72640375b"} Oct 30 19:15:16.258681 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:16.258624 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-c3a5b9-29889795-dz7tn" podStartSLOduration=16.258610302 podStartE2EDuration="16.258610302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:16.253113503 +0000 UTC m=+2483.782144640" watchObservedRunningTime="2026-10-30 19:15:16.258610302 +0000 UTC m=+2483.787641439" Oct 30 19:15:16.926920 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:16.926822 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:17.239110 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:17.239031 6960 generic.go:356] "Generic (PLEG): container finished" podID="c13c7df4-97b2-4a0d-855b-049561541df1" containerID="e458bf289428c91466fb3cd987cf7d1ceb1e4a2fb867f122273d79d72640375b" exitCode=60 Oct 30 19:15:17.239110 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:17.239071 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-c3a5b9-29889795-dz7tn" event={"ID":"c13c7df4-97b2-4a0d-855b-049561541df1","Type":"ContainerDied","Data":"e458bf289428c91466fb3cd987cf7d1ceb1e4a2fb867f122273d79d72640375b"} Oct 30 19:15:17.239347 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:17.239213 6960 scope.go:122] "RemoveContainer" containerID="5633f35b8513549165aac0b8c8a5205317c5683e49b7a619855342d68b0ee3cd" Oct 30 19:15:17.239621 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:17.239602 6960 scope.go:122] "RemoveContainer" containerID="e458bf289428c91466fb3cd987cf7d1ceb1e4a2fb867f122273d79d72640375b" Oct 30 19:15:17.240031 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:15:17.240006 6960 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-c3a5b9-29889795-dz7tn_odh-ai-gateway-infra(c13c7df4-97b2-4a0d-855b-049561541df1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-c3a5b9-29889795-dz7tn" podUID="c13c7df4-97b2-4a0d-855b-049561541df1" Oct 30 19:15:17.274156 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:17.274121 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-c3a5b9-29889795-dz7tn"] Oct 30 19:15:17.837018 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:17.836978 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:18.423373 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:18.423351 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-c3a5b9-29889795-dz7tn" Oct 30 19:15:18.497546 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:18.497520 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c13c7df4-97b2-4a0d-855b-049561541df1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c13c7df4-97b2-4a0d-855b-049561541df1-maas-api-service-ca\") pod \"c13c7df4-97b2-4a0d-855b-049561541df1\" (UID: \"c13c7df4-97b2-4a0d-855b-049561541df1\") " Oct 30 19:15:18.497716 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:18.497595 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c13c7df4-97b2-4a0d-855b-049561541df1-kube-api-access-k8kqz\" (UniqueName: \"kubernetes.io/projected/c13c7df4-97b2-4a0d-855b-049561541df1-kube-api-access-k8kqz\") pod \"c13c7df4-97b2-4a0d-855b-049561541df1\" (UID: \"c13c7df4-97b2-4a0d-855b-049561541df1\") " Oct 30 19:15:18.497988 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:18.497930 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c13c7df4-97b2-4a0d-855b-049561541df1-maas-api-service-ca" pod "c13c7df4-97b2-4a0d-855b-049561541df1" (UID: "c13c7df4-97b2-4a0d-855b-049561541df1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:15:18.498401 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:18.498345 6960 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c13c7df4-97b2-4a0d-855b-049561541df1-maas-api-service-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:18.500794 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:18.500769 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c13c7df4-97b2-4a0d-855b-049561541df1-kube-api-access-k8kqz" pod "c13c7df4-97b2-4a0d-855b-049561541df1" (UID: "c13c7df4-97b2-4a0d-855b-049561541df1"). InnerVolumeSpecName "kube-api-access-k8kqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:15:18.600429 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:18.600388 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8kqz\" (UniqueName: \"kubernetes.io/projected/c13c7df4-97b2-4a0d-855b-049561541df1-kube-api-access-k8kqz\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:15:18.808437 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:18.808404 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:19.256230 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:19.256135 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-c3a5b9-29889795-dz7tn" Oct 30 19:15:19.256230 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:19.256177 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-c3a5b9-29889795-dz7tn" event={"ID":"c13c7df4-97b2-4a0d-855b-049561541df1","Type":"ContainerDied","Data":"1cd45ebf93ac34b88c24a13cc4fe8726665bbf72ff610e47a52dfb45f610840f"} Oct 30 19:15:19.256495 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:19.256248 6960 scope.go:122] "RemoveContainer" containerID="e458bf289428c91466fb3cd987cf7d1ceb1e4a2fb867f122273d79d72640375b" Oct 30 19:15:19.283708 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:19.283686 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-c3a5b9-29889795-dz7tn"] Oct 30 19:15:19.288897 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:19.288868 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-c3a5b9-29889795-dz7tn"] Oct 30 19:15:19.840690 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:19.840655 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:20.594895 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:20.594866 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c13c7df4-97b2-4a0d-855b-049561541df1" path="/var/lib/kubelet/pods/c13c7df4-97b2-4a0d-855b-049561541df1/volumes" Oct 30 19:15:20.854048 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:20.853963 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:21.813755 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:21.813724 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:22.953173 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:22.953119 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:23.941245 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:15:23.941211 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 19:15:24.409635 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:24.409602 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:25.600761 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:25.600725 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:26.477546 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:26.477502 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:27.322515 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:27.322481 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:28.245379 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:28.245345 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:29.199937 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:29.199854 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:30.041221 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:30.041139 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:31.018919 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:31.018884 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:31.799759 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:31.799711 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:32.637634 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:32.637604 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:33.647830 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:33.647780 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:34.482865 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:34.482827 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:35.296387 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:35.296356 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:36.193113 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:36.193067 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:37.013324 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:37.013286 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:37.862024 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:37.861989 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:38.834031 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:38.833995 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:39.744306 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:39.744270 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:40.618280 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:40.618248 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:41.798266 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:41.798234 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:42.762720 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:42.762614 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:43.788187 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:43.788132 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:44.625084 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:44.625034 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:45.530875 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:45.530829 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:46.326568 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:46.326535 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:47.359099 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:47.359059 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:48.136913 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:48.136879 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:49.017767 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:49.017721 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:49.829230 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:49.829196 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:50.848694 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:50.848656 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:51.961658 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:51.961616 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:52.875715 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:52.875290 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:53.893173 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:53.893126 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:53.960306 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:15:53.960197 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 19:15:54.163310 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:54.163237 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:54.354192 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:54.354137 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:54.843377 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:54.843325 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:55.816338 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:55.816300 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:56.600852 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:56.600820 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:57.423529 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:57.423489 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:58.462733 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:58.462554 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:15:59.518494 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:15:59.518457 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:00.316601 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:16:00.316566 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:01.260734 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:16:01.260701 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:01.511839 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:16:01.511753 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:01.701775 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:16:01.701735 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:02.469644 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:16:02.469599 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:02.596766 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:16:02.596698 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:03.415878 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:16:03.415791 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:04.304313 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:16:04.304210 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:05.111111 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:16:05.111072 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:05.857396 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:16:05.857356 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:05.989927 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:16:05.989891 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:06.899449 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:16:06.899415 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:07.693872 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:16:07.693838 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:08.670197 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:16:08.670086 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:09.639774 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:16:09.639731 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:10.559406 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:16:10.559374 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:11.603278 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:16:11.603242 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:12.822039 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:16:12.822004 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:14.332781 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:16:14.332749 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:14.457066 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:16:14.457032 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:15.301587 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:16:15.301537 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:16.242049 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:16:16.242011 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:17.238783 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:16:17.238746 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:18.163636 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:16:18.163601 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:18.995402 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:16:18.995369 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:19.824386 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:16:19.824347 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:20.614242 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:16:20.614205 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:21.463355 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:16:21.463318 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:22.338421 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:16:22.338363 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:23.206985 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:16:23.206950 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:23.936716 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:16:23.936680 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 19:16:24.077566 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:16:24.077516 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:24.930360 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:16:24.930323 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:25.121290 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:16:25.121238 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:25.241594 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:16:25.241515 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:25.698278 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:16:25.698245 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:26.613907 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:16:26.613875 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:27.430918 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:16:27.430884 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:28.299936 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:16:28.299885 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:29.083196 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:16:29.083074 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:30.035678 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:16:30.035641 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:30.810635 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:16:30.810597 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:31.483210 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:16:31.483124 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:31.598187 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:16:31.598138 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:31.722723 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:16:31.722663 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:32.491018 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:16:32.490982 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:33.317894 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:16:33.317842 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:34.175904 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:16:34.175867 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:35.007437 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:16:35.007403 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:35.941264 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:16:35.941230 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:36.894611 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:16:36.894578 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:37.754375 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:16:37.753990 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:38.855621 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:16:38.855581 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:39.726523 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:16:39.726494 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:40.591744 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:16:40.591679 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:41.531008 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:16:41.530970 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:42.433904 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:16:42.433871 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:43.270926 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:16:43.270868 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:44.165271 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:16:44.165235 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:45.109205 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:16:45.109165 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:45.903750 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:16:45.903719 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:46.709508 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:16:46.709467 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:47.614935 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:16:47.614898 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:48.408949 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:16:48.408915 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:49.307646 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:16:49.307613 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:50.267700 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:16:50.267670 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:51.293624 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:16:51.293585 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:52.257118 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:16:52.255698 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:53.362464 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:16:53.362419 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:53.933979 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:16:53.933944 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 19:16:54.226458 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:16:54.226364 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:54.353435 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:16:54.353397 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:55.243679 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:16:55.243637 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:56.016764 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:16:56.016730 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:56.863168 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:16:56.862918 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:57.825427 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:16:57.825398 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:58.615790 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:16:58.615730 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:16:59.420110 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:16:59.420075 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:00.229817 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:00.229757 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:01.216445 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:01.216409 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:02.158575 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:02.158534 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:03.047281 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:03.047245 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:04.272729 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:04.272691 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:05.161336 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:05.161300 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:06.033765 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:06.033726 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:06.193001 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:06.192808 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:06.941518 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:06.941483 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:07.947462 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:07.947433 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:08.836291 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:08.836232 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:09.627875 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:09.627844 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:10.544922 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:10.544891 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:11.477720 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:11.477687 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:12.248834 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:12.248796 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:13.228023 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:13.227984 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:14.051866 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:14.051826 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:15.073095 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:15.073044 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:16.093243 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:16.093203 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:16.331664 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:16.331630 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:16.476177 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:16.476057 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:17.337704 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:17.337674 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:18.326762 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:18.326720 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:19.164883 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:19.164839 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:20.108534 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:20.108483 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:20.922266 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:20.922229 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:21.751931 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:21.751895 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:22.696756 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:22.696720 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:23.490204 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:23.490161 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:23.934743 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:17:23.934703 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 19:17:24.419325 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:24.419296 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:25.279256 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:25.279216 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:26.237349 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:26.237306 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:27.198599 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:27.198572 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:28.047319 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:28.047285 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:28.369210 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:28.369174 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:28.797044 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:28.796970 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:29.205030 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:29.205000 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:30.093013 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:30.092978 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:30.876688 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:30.876599 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:31.647838 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:31.647784 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:32.544979 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:32.544926 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:33.711038 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:33.710927 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:34.915982 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:34.915948 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:35.608016 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:35.607986 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:35.847087 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:35.847050 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:35.979851 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:35.979767 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:36.112484 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:36.112455 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:36.595362 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:36.595323 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:36.792078 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:36.792049 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:36.971571 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:36.971482 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:37.118337 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:37.118294 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:38.175396 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:38.175368 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-xxv5m"] Oct 30 19:17:38.175817 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:38.175598 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-xxv5m" podUID="71138058-4eed-4e10-b831-50aa2d950f79" containerName="istio-proxy" containerID="cri-o://bdd65456188b4c11e5ee67c0131589a35888bea6e2be6ad3ed298ef02590579e" gracePeriod=30 Oct 30 19:17:38.361319 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:38.361223 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:38.522906 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:38.522819 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:39.116003 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:39.115969 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:39.803350 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:39.803311 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:40.750361 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:40.750327 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:41.258368 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:41.258330 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:41.903195 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:41.903162 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:42.919122 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:42.919084 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:43.457511 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:43.457453 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-xxv5m" Oct 30 19:17:43.574064 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:43.574031 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/71138058-4eed-4e10-b831-50aa2d950f79-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71138058-4eed-4e10-b831-50aa2d950f79-istio-podinfo\") pod \"71138058-4eed-4e10-b831-50aa2d950f79\" (UID: \"71138058-4eed-4e10-b831-50aa2d950f79\") " Oct 30 19:17:43.574273 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:43.574252 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71138058-4eed-4e10-b831-50aa2d950f79-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71138058-4eed-4e10-b831-50aa2d950f79-workload-socket\") pod \"71138058-4eed-4e10-b831-50aa2d950f79\" (UID: \"71138058-4eed-4e10-b831-50aa2d950f79\") " Oct 30 19:17:43.574362 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:43.574347 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71138058-4eed-4e10-b831-50aa2d950f79-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71138058-4eed-4e10-b831-50aa2d950f79-istio-data\") pod \"71138058-4eed-4e10-b831-50aa2d950f79\" (UID: \"71138058-4eed-4e10-b831-50aa2d950f79\") " Oct 30 19:17:43.574429 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:43.574414 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71138058-4eed-4e10-b831-50aa2d950f79-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71138058-4eed-4e10-b831-50aa2d950f79-credential-socket\") pod \"71138058-4eed-4e10-b831-50aa2d950f79\" (UID: \"71138058-4eed-4e10-b831-50aa2d950f79\") " Oct 30 19:17:43.574494 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:43.574478 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71138058-4eed-4e10-b831-50aa2d950f79-istio-token\" (UniqueName: \"kubernetes.io/projected/71138058-4eed-4e10-b831-50aa2d950f79-istio-token\") pod \"71138058-4eed-4e10-b831-50aa2d950f79\" (UID: \"71138058-4eed-4e10-b831-50aa2d950f79\") " Oct 30 19:17:43.574554 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:43.574509 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71138058-4eed-4e10-b831-50aa2d950f79-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71138058-4eed-4e10-b831-50aa2d950f79-workload-certs\") pod \"71138058-4eed-4e10-b831-50aa2d950f79\" (UID: \"71138058-4eed-4e10-b831-50aa2d950f79\") " Oct 30 19:17:43.574554 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:43.574536 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71138058-4eed-4e10-b831-50aa2d950f79-kube-api-access-4bgg6\" (UniqueName: \"kubernetes.io/projected/71138058-4eed-4e10-b831-50aa2d950f79-kube-api-access-4bgg6\") pod \"71138058-4eed-4e10-b831-50aa2d950f79\" (UID: \"71138058-4eed-4e10-b831-50aa2d950f79\") " Oct 30 19:17:43.574670 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:43.574549 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71138058-4eed-4e10-b831-50aa2d950f79-workload-socket" pod "71138058-4eed-4e10-b831-50aa2d950f79" (UID: "71138058-4eed-4e10-b831-50aa2d950f79"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:17:43.574670 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:43.574606 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71138058-4eed-4e10-b831-50aa2d950f79-istio-data" pod "71138058-4eed-4e10-b831-50aa2d950f79" (UID: "71138058-4eed-4e10-b831-50aa2d950f79"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:17:43.574782 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:43.574707 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71138058-4eed-4e10-b831-50aa2d950f79-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71138058-4eed-4e10-b831-50aa2d950f79-istio-envoy\") pod \"71138058-4eed-4e10-b831-50aa2d950f79\" (UID: \"71138058-4eed-4e10-b831-50aa2d950f79\") " Oct 30 19:17:43.574782 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:43.574708 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71138058-4eed-4e10-b831-50aa2d950f79-credential-socket" pod "71138058-4eed-4e10-b831-50aa2d950f79" (UID: "71138058-4eed-4e10-b831-50aa2d950f79"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:17:43.574782 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:43.574753 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71138058-4eed-4e10-b831-50aa2d950f79-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71138058-4eed-4e10-b831-50aa2d950f79-istiod-ca-cert\") pod \"71138058-4eed-4e10-b831-50aa2d950f79\" (UID: \"71138058-4eed-4e10-b831-50aa2d950f79\") " Oct 30 19:17:43.575029 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:43.574787 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71138058-4eed-4e10-b831-50aa2d950f79-workload-certs" pod "71138058-4eed-4e10-b831-50aa2d950f79" (UID: "71138058-4eed-4e10-b831-50aa2d950f79"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:17:43.575116 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:43.575092 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71138058-4eed-4e10-b831-50aa2d950f79-istiod-ca-cert" pod "71138058-4eed-4e10-b831-50aa2d950f79" (UID: "71138058-4eed-4e10-b831-50aa2d950f79"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:17:43.575864 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:43.575843 6960 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71138058-4eed-4e10-b831-50aa2d950f79-workload-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:43.575921 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:43.575877 6960 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71138058-4eed-4e10-b831-50aa2d950f79-istio-data\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:43.575921 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:43.575891 6960 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71138058-4eed-4e10-b831-50aa2d950f79-credential-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:43.575921 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:43.575905 6960 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71138058-4eed-4e10-b831-50aa2d950f79-workload-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:43.575921 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:43.575919 6960 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71138058-4eed-4e10-b831-50aa2d950f79-istiod-ca-cert\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:43.576690 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:43.576669 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/71138058-4eed-4e10-b831-50aa2d950f79-istio-podinfo" pod "71138058-4eed-4e10-b831-50aa2d950f79" (UID: "71138058-4eed-4e10-b831-50aa2d950f79"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:17:43.576782 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:43.576752 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71138058-4eed-4e10-b831-50aa2d950f79-istio-token" pod "71138058-4eed-4e10-b831-50aa2d950f79" (UID: "71138058-4eed-4e10-b831-50aa2d950f79"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:17:43.577367 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:43.577344 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71138058-4eed-4e10-b831-50aa2d950f79-istio-envoy" pod "71138058-4eed-4e10-b831-50aa2d950f79" (UID: "71138058-4eed-4e10-b831-50aa2d950f79"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:17:43.577494 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:43.577374 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71138058-4eed-4e10-b831-50aa2d950f79-kube-api-access-4bgg6" pod "71138058-4eed-4e10-b831-50aa2d950f79" (UID: "71138058-4eed-4e10-b831-50aa2d950f79"). InnerVolumeSpecName "kube-api-access-4bgg6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:17:43.678181 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:43.678123 6960 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71138058-4eed-4e10-b831-50aa2d950f79-istio-token\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:43.678181 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:43.678180 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bgg6\" (UniqueName: \"kubernetes.io/projected/71138058-4eed-4e10-b831-50aa2d950f79-kube-api-access-4bgg6\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:43.678398 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:43.678198 6960 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71138058-4eed-4e10-b831-50aa2d950f79-istio-envoy\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:43.678398 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:43.678214 6960 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71138058-4eed-4e10-b831-50aa2d950f79-istio-podinfo\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:17:43.700309 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:43.700273 6960 generic.go:356] "Generic (PLEG): container finished" podID="71138058-4eed-4e10-b831-50aa2d950f79" containerID="bdd65456188b4c11e5ee67c0131589a35888bea6e2be6ad3ed298ef02590579e" exitCode=0 Oct 30 19:17:43.700499 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:43.700323 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-xxv5m" Oct 30 19:17:43.700499 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:43.700351 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-xxv5m" event={"ID":"71138058-4eed-4e10-b831-50aa2d950f79","Type":"ContainerDied","Data":"bdd65456188b4c11e5ee67c0131589a35888bea6e2be6ad3ed298ef02590579e"} Oct 30 19:17:43.700499 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:43.700410 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-xxv5m" event={"ID":"71138058-4eed-4e10-b831-50aa2d950f79","Type":"ContainerDied","Data":"8a43e94145eef632458063af97b9a2e4b071e3ba9c19abd6df91bb46467d256e"} Oct 30 19:17:43.700499 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:43.700437 6960 scope.go:122] "RemoveContainer" containerID="bdd65456188b4c11e5ee67c0131589a35888bea6e2be6ad3ed298ef02590579e" Oct 30 19:17:43.712726 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:43.712707 6960 scope.go:122] "RemoveContainer" containerID="bdd65456188b4c11e5ee67c0131589a35888bea6e2be6ad3ed298ef02590579e" Oct 30 19:17:43.713051 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:17:43.713028 6960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdd65456188b4c11e5ee67c0131589a35888bea6e2be6ad3ed298ef02590579e\": container with ID starting with bdd65456188b4c11e5ee67c0131589a35888bea6e2be6ad3ed298ef02590579e not found: ID does not exist" containerID="bdd65456188b4c11e5ee67c0131589a35888bea6e2be6ad3ed298ef02590579e" Oct 30 19:17:43.713102 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:43.713062 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdd65456188b4c11e5ee67c0131589a35888bea6e2be6ad3ed298ef02590579e"} err="failed to get container status \"bdd65456188b4c11e5ee67c0131589a35888bea6e2be6ad3ed298ef02590579e\": rpc error: code = NotFound desc = could not find container \"bdd65456188b4c11e5ee67c0131589a35888bea6e2be6ad3ed298ef02590579e\": container with ID starting with bdd65456188b4c11e5ee67c0131589a35888bea6e2be6ad3ed298ef02590579e not found: ID does not exist" Oct 30 19:17:43.741094 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:43.741070 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-xxv5m"] Oct 30 19:17:43.749777 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:43.749755 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-xxv5m"] Oct 30 19:17:43.884188 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:43.884067 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:44.342114 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:44.342074 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:44.554180 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:44.554009 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:44.733219 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:44.728895 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71138058-4eed-4e10-b831-50aa2d950f79" path="/var/lib/kubelet/pods/71138058-4eed-4e10-b831-50aa2d950f79/volumes" Oct 30 19:17:45.427675 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:45.427642 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:45.766000 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:45.765924 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-585dmj7jc"] Oct 30 19:17:45.766697 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:45.766683 6960 cpu_manager.go:414] "containerMap: removing container" podUID="71138058-4eed-4e10-b831-50aa2d950f79" containerName="istio-proxy" Oct 30 19:17:45.766753 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:45.766700 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71138058-4eed-4e10-b831-50aa2d950f79" containerName="istio-proxy" Oct 30 19:17:45.766753 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:45.766709 6960 cpu_manager.go:414] "containerMap: removing container" podUID="c13c7df4-97b2-4a0d-855b-049561541df1" containerName="cleanup" Oct 30 19:17:45.766753 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:45.766714 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c13c7df4-97b2-4a0d-855b-049561541df1" containerName="cleanup" Oct 30 19:17:45.766753 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:45.766741 6960 cpu_manager.go:414] "containerMap: removing container" podUID="c13c7df4-97b2-4a0d-855b-049561541df1" containerName="cleanup" Oct 30 19:17:45.766753 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:45.766746 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c13c7df4-97b2-4a0d-855b-049561541df1" containerName="cleanup" Oct 30 19:17:45.766930 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:45.766906 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="71138058-4eed-4e10-b831-50aa2d950f79" containerName="istio-proxy" Oct 30 19:17:45.766930 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:45.766915 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="c13c7df4-97b2-4a0d-855b-049561541df1" containerName="cleanup" Oct 30 19:17:45.766930 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:45.766926 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="c13c7df4-97b2-4a0d-855b-049561541df1" containerName="cleanup" Oct 30 19:17:45.771218 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:45.771195 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-585dmj7jc" Oct 30 19:17:45.776219 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:45.776195 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-21d896\"/\"payload-processing-external-model-plugins-e2e-extmod-a-21d896\"" Oct 30 19:17:45.776570 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:45.776549 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-21d896\"/\"agc-cred-676f802b0d13\"" Oct 30 19:17:45.776679 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:45.776656 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-21d896\"/\"agc-cred-1b4803c0417a\"" Oct 30 19:17:45.782646 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:45.781307 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-21d896\"/\"payload-processing-external-model-e2e--3f19eb37-dockercfg-phgbc\"" Oct 30 19:17:45.788064 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:45.788040 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-585dmj7jc"] Oct 30 19:17:45.797605 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:45.796064 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-21d896\"/\"openshift-service-ca.crt\"" Oct 30 19:17:45.797605 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:45.796358 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-21d896\"/\"kube-root-ca.crt\"" Oct 30 19:17:45.797605 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:45.796394 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-21d896\"/\"routing-overlay\"" Oct 30 19:17:45.840110 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:45.840079 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21b44f0f-4b22-4369-9045-c9065515f816-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-21d896-585dmj7jc\" (UID: \"21b44f0f-4b22-4369-9045-c9065515f816\") " pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-585dmj7jc" Oct 30 19:17:45.840300 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:45.840283 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/21b44f0f-4b22-4369-9045-c9065515f816-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-21d896-585dmj7jc\" (UID: \"21b44f0f-4b22-4369-9045-c9065515f816\") " pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-585dmj7jc" Oct 30 19:17:45.840365 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:45.840314 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m46n\" (UniqueName: \"kubernetes.io/projected/21b44f0f-4b22-4369-9045-c9065515f816-kube-api-access-2m46n\") pod \"payload-processing-external-model-e2e-extmod-a-21d896-585dmj7jc\" (UID: \"21b44f0f-4b22-4369-9045-c9065515f816\") " pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-585dmj7jc" Oct 30 19:17:45.840497 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:45.840477 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/21b44f0f-4b22-4369-9045-c9065515f816-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-21d896-585dmj7jc\" (UID: \"21b44f0f-4b22-4369-9045-c9065515f816\") " pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-585dmj7jc" Oct 30 19:17:45.941924 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:45.941888 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/21b44f0f-4b22-4369-9045-c9065515f816-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-21d896-585dmj7jc\" (UID: \"21b44f0f-4b22-4369-9045-c9065515f816\") " pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-585dmj7jc" Oct 30 19:17:45.942131 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:45.942003 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21b44f0f-4b22-4369-9045-c9065515f816-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-21d896-585dmj7jc\" (UID: \"21b44f0f-4b22-4369-9045-c9065515f816\") " pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-585dmj7jc" Oct 30 19:17:45.942317 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:45.942294 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/21b44f0f-4b22-4369-9045-c9065515f816-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-21d896-585dmj7jc\" (UID: \"21b44f0f-4b22-4369-9045-c9065515f816\") " pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-585dmj7jc" Oct 30 19:17:45.942414 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:45.942343 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m46n\" (UniqueName: \"kubernetes.io/projected/21b44f0f-4b22-4369-9045-c9065515f816-kube-api-access-2m46n\") pod \"payload-processing-external-model-e2e-extmod-a-21d896-585dmj7jc\" (UID: \"21b44f0f-4b22-4369-9045-c9065515f816\") " pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-585dmj7jc" Oct 30 19:17:45.942681 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:45.942657 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/21b44f0f-4b22-4369-9045-c9065515f816-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-21d896-585dmj7jc\" (UID: \"21b44f0f-4b22-4369-9045-c9065515f816\") " pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-585dmj7jc" Oct 30 19:17:45.942877 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:45.942848 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21b44f0f-4b22-4369-9045-c9065515f816-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-21d896-585dmj7jc\" (UID: \"21b44f0f-4b22-4369-9045-c9065515f816\") " pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-585dmj7jc" Oct 30 19:17:45.944992 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:45.944968 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/21b44f0f-4b22-4369-9045-c9065515f816-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-21d896-585dmj7jc\" (UID: \"21b44f0f-4b22-4369-9045-c9065515f816\") " pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-585dmj7jc" Oct 30 19:17:45.957679 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:45.957659 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m46n\" (UniqueName: \"kubernetes.io/projected/21b44f0f-4b22-4369-9045-c9065515f816-kube-api-access-2m46n\") pod \"payload-processing-external-model-e2e-extmod-a-21d896-585dmj7jc\" (UID: \"21b44f0f-4b22-4369-9045-c9065515f816\") " pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-585dmj7jc" Oct 30 19:17:46.087725 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:46.087698 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-585dmj7jc" Oct 30 19:17:46.390676 ip-10-0-140-163 kubenswrapper[6960]: W1030 19:17:46.390645 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21b44f0f_4b22_4369_9045_c9065515f816.slice/crio-89fe98e080941b9d933a99c93d129218fee1ada84fbc1db2c20494f672d43031 WatchSource:0}: Error finding container 89fe98e080941b9d933a99c93d129218fee1ada84fbc1db2c20494f672d43031: Status 404 returned error can't find the container with id 89fe98e080941b9d933a99c93d129218fee1ada84fbc1db2c20494f672d43031 Oct 30 19:17:46.391384 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:46.391358 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-585dmj7jc"] Oct 30 19:17:46.473185 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:46.472410 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:46.836061 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:46.836024 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-585dmj7jc" event={"ID":"21b44f0f-4b22-4369-9045-c9065515f816","Type":"ContainerStarted","Data":"7bc156f4fa01301450cb1c9e4ce5e56eacf3c3a17297d79841e3c885b15b4be3"} Oct 30 19:17:46.836302 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:46.836093 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-585dmj7jc" Oct 30 19:17:46.836302 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:46.836117 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-585dmj7jc" event={"ID":"21b44f0f-4b22-4369-9045-c9065515f816","Type":"ContainerStarted","Data":"89fe98e080941b9d933a99c93d129218fee1ada84fbc1db2c20494f672d43031"} Oct 30 19:17:46.859114 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:46.859034 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-585dmj7jc" podStartSLOduration=1.859020108 podStartE2EDuration="1.859020108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:17:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:17:46.854629662 +0000 UTC m=+2634.383660801" watchObservedRunningTime="2026-10-30 19:17:46.859020108 +0000 UTC m=+2634.388051246" Oct 30 19:17:47.427589 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:47.427553 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:48.228381 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:48.228354 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:49.004344 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:49.004312 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:49.863497 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:49.863460 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:50.710726 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:50.710689 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:51.746571 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:51.746537 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:52.888442 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:52.888407 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:53.706304 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:53.706269 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:53.933572 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:17:53.933490 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 19:17:54.262930 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:54.262803 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:54.583976 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:54.582469 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:55.428561 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:55.428521 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:56.385360 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:56.385328 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:57.651702 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:57.651662 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:57.845541 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:57.845512 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-585dmj7jc" Oct 30 19:17:58.837271 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:58.837239 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:17:59.860608 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:17:59.860575 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:00.639680 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:00.639642 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:01.426819 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:01.426784 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:02.470035 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:02.470000 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:03.417822 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:03.417769 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:04.334401 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:04.334367 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:05.192338 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:05.192302 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:06.135965 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:06.135928 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:07.020648 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:07.020612 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:07.892201 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:07.892169 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:08.842800 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:08.842742 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:09.709350 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:09.709310 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:10.524989 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:10.524954 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:11.412250 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:11.412209 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:12.238137 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:12.238102 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:13.126979 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:13.126937 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:13.905214 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:13.905184 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:14.696204 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:14.696114 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:15.623608 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:15.623572 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:16.404168 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:16.404115 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:17.258505 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:17.258467 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:18.211600 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:18.211565 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:19.009649 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:19.009609 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:19.770218 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:19.770177 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:20.740489 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:20.740450 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:21.720277 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:21.720245 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:22.526188 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:22.526159 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:23.390660 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:23.390623 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:23.936636 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:18:23.936594 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 19:18:24.237774 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:24.237670 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:24.658130 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:24.658075 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:25.109728 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:25.109693 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:26.126226 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:26.126195 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:26.924472 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:26.924429 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:28.061686 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:28.061652 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:29.039980 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:29.039944 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:30.215876 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:30.215824 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:31.221232 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:31.220430 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:32.184137 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:32.184099 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:33.092170 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:33.091954 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:33.857302 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:33.857267 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:34.934121 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:34.934078 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:36.021542 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:36.021499 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:37.100498 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:37.100463 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:38.046346 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:38.046314 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:39.092763 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:39.092721 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:40.071586 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:40.071557 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:40.939476 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:40.939436 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:41.916077 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:41.916038 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:42.744829 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:42.744795 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:43.680293 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:43.680255 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:44.479417 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:44.479378 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:45.410521 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:45.410484 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:46.023070 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:46.023025 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:46.192781 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:46.192740 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:46.928029 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:46.927989 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:47.403231 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:47.396619 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:47.523134 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:47.523092 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:47.959512 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:47.959480 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:48.097871 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:48.097834 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:48.292971 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:48.292890 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:48.423050 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:48.422598 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:48.588732 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:48.588701 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:48.747703 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:48.747665 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:48.989373 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:48.989214 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:49.786069 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:49.786035 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:50.766886 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:50.766834 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:51.936763 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:51.936722 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:53.005166 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:53.000860 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:53.804890 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:53.804844 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:53.937886 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:18:53.937849 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 19:18:54.236905 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:54.236833 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-extmod-shr-user-cce8b2/e2e-usage-sim-4bc3fa-6677768f7d-b5tnm"] Oct 30 19:18:54.237647 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:54.237631 6960 cpu_manager.go:414] "containerMap: removing container" podUID="c13c7df4-97b2-4a0d-855b-049561541df1" containerName="cleanup" Oct 30 19:18:54.237699 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:54.237649 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c13c7df4-97b2-4a0d-855b-049561541df1" containerName="cleanup" Oct 30 19:18:54.237761 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:54.237752 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="c13c7df4-97b2-4a0d-855b-049561541df1" containerName="cleanup" Oct 30 19:18:54.241354 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:54.241336 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-cce8b2/e2e-usage-sim-4bc3fa-6677768f7d-b5tnm" Oct 30 19:18:54.255173 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:54.255074 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-extmod-shr-user-cce8b2\"/\"openshift-service-ca.crt\"" Oct 30 19:18:54.255173 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:54.255085 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-extmod-shr-user-cce8b2\"/\"kube-root-ca.crt\"" Oct 30 19:18:54.255173 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:54.255127 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-extmod-shr-user-cce8b2\"/\"e2e-usage-sim-4bc3fa-cert\"" Oct 30 19:18:54.255426 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:54.255305 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-extmod-shr-user-cce8b2\"/\"default-dockercfg-85d6r\"" Oct 30 19:18:54.256235 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:54.256217 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-extmod-shr-user-cce8b2/e2e-usage-sim-4bc3fa-6677768f7d-b5tnm"] Oct 30 19:18:54.300398 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:54.300360 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:54.356473 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:54.356441 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrx69\" (UniqueName: \"kubernetes.io/projected/8e4832f5-ee56-4ea9-82c2-9fffed013c35-kube-api-access-hrx69\") pod \"e2e-usage-sim-4bc3fa-6677768f7d-b5tnm\" (UID: \"8e4832f5-ee56-4ea9-82c2-9fffed013c35\") " pod="e2e-extmod-shr-user-cce8b2/e2e-usage-sim-4bc3fa-6677768f7d-b5tnm" Oct 30 19:18:54.356675 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:54.356654 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/8e4832f5-ee56-4ea9-82c2-9fffed013c35-tls\") pod \"e2e-usage-sim-4bc3fa-6677768f7d-b5tnm\" (UID: \"8e4832f5-ee56-4ea9-82c2-9fffed013c35\") " pod="e2e-extmod-shr-user-cce8b2/e2e-usage-sim-4bc3fa-6677768f7d-b5tnm" Oct 30 19:18:54.457952 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:54.457920 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/8e4832f5-ee56-4ea9-82c2-9fffed013c35-tls\") pod \"e2e-usage-sim-4bc3fa-6677768f7d-b5tnm\" (UID: \"8e4832f5-ee56-4ea9-82c2-9fffed013c35\") " pod="e2e-extmod-shr-user-cce8b2/e2e-usage-sim-4bc3fa-6677768f7d-b5tnm" Oct 30 19:18:54.458128 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:54.458069 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrx69\" (UniqueName: \"kubernetes.io/projected/8e4832f5-ee56-4ea9-82c2-9fffed013c35-kube-api-access-hrx69\") pod \"e2e-usage-sim-4bc3fa-6677768f7d-b5tnm\" (UID: \"8e4832f5-ee56-4ea9-82c2-9fffed013c35\") " pod="e2e-extmod-shr-user-cce8b2/e2e-usage-sim-4bc3fa-6677768f7d-b5tnm" Oct 30 19:18:54.460614 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:54.460586 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/8e4832f5-ee56-4ea9-82c2-9fffed013c35-tls\") pod \"e2e-usage-sim-4bc3fa-6677768f7d-b5tnm\" (UID: \"8e4832f5-ee56-4ea9-82c2-9fffed013c35\") " pod="e2e-extmod-shr-user-cce8b2/e2e-usage-sim-4bc3fa-6677768f7d-b5tnm" Oct 30 19:18:54.470381 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:54.470356 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrx69\" (UniqueName: \"kubernetes.io/projected/8e4832f5-ee56-4ea9-82c2-9fffed013c35-kube-api-access-hrx69\") pod \"e2e-usage-sim-4bc3fa-6677768f7d-b5tnm\" (UID: \"8e4832f5-ee56-4ea9-82c2-9fffed013c35\") " pod="e2e-extmod-shr-user-cce8b2/e2e-usage-sim-4bc3fa-6677768f7d-b5tnm" Oct 30 19:18:54.555344 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:54.555318 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-cce8b2/e2e-usage-sim-4bc3fa-6677768f7d-b5tnm" Oct 30 19:18:54.645181 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:54.644993 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:54.842783 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:54.842755 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-extmod-shr-user-cce8b2/e2e-usage-sim-4bc3fa-6677768f7d-b5tnm"] Oct 30 19:18:54.843099 ip-10-0-140-163 kubenswrapper[6960]: W1030 19:18:54.843078 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8e4832f5_ee56_4ea9_82c2_9fffed013c35.slice/crio-fcb2ad78970574c728c00c8b713ee5757118090dc95259e6fcc88d2da27770fc WatchSource:0}: Error finding container fcb2ad78970574c728c00c8b713ee5757118090dc95259e6fcc88d2da27770fc: Status 404 returned error can't find the container with id fcb2ad78970574c728c00c8b713ee5757118090dc95259e6fcc88d2da27770fc Oct 30 19:18:55.502614 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:55.502574 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-cce8b2/e2e-usage-sim-4bc3fa-6677768f7d-b5tnm" event={"ID":"8e4832f5-ee56-4ea9-82c2-9fffed013c35","Type":"ContainerStarted","Data":"8fe7c236ca19c2856395cb6c8979b6a230699b2a90d827f8417d49f45c82ffa7"} Oct 30 19:18:55.503023 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:55.502624 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-cce8b2/e2e-usage-sim-4bc3fa-6677768f7d-b5tnm" event={"ID":"8e4832f5-ee56-4ea9-82c2-9fffed013c35","Type":"ContainerStarted","Data":"fcb2ad78970574c728c00c8b713ee5757118090dc95259e6fcc88d2da27770fc"} Oct 30 19:18:55.503023 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:55.502640 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-extmod-shr-user-cce8b2/e2e-usage-sim-4bc3fa-6677768f7d-b5tnm" Oct 30 19:18:55.508075 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:55.508050 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-extmod-shr-user-cce8b2/e2e-usage-sim-4bc3fa-6677768f7d-b5tnm" Oct 30 19:18:55.520193 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:55.520127 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-extmod-shr-user-cce8b2/e2e-usage-sim-4bc3fa-6677768f7d-b5tnm" podStartSLOduration=1.5201111680000001 podStartE2EDuration="1.520111168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:18:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:18:55.519026129 +0000 UTC m=+2703.048057290" watchObservedRunningTime="2026-10-30 19:18:55.520111168 +0000 UTC m=+2703.049142306" Oct 30 19:18:55.614509 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:55.614477 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:55.875835 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:55.875792 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:56.000709 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:56.000651 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:57.509775 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:57.509739 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:58.321746 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:58.321712 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:58.428010 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:58.427973 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:58.835540 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:58.835507 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:58.947821 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:58.947787 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:59.812309 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:59.812274 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:18:59.935103 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:18:59.935068 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:00.059167 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:00.058883 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:00.489191 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:00.482091 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:00.643101 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:00.643065 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:00.826401 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:00.826370 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:01.257999 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:01.257908 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:01.385726 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:01.385692 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:01.507500 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:01.507463 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:01.795679 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:01.795626 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:03.045330 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:03.045296 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:03.655117 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:03.655042 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:03.971484 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:03.971384 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:04.117322 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:04.117284 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:04.743513 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:04.743448 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:04.896170 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:04.887496 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:05.936013 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:05.935982 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:06.913289 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:06.913235 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:07.285544 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:07.285381 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:07.457814 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:07.457758 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:08.133676 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:08.133626 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:08.249069 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:08.248572 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:08.460222 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:08.459797 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:08.720468 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:08.719228 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:08.986687 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:08.986591 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-79b45db86b-fj8wf"] Oct 30 19:19:08.993430 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:08.993406 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-79b45db86b-fj8wf" Oct 30 19:19:09.000715 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:09.000596 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Oct 30 19:19:09.001094 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:09.001068 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Oct 30 19:19:09.001268 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:09.001185 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Oct 30 19:19:09.001268 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:09.001229 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-acfdea0a628c\"" Oct 30 19:19:09.001408 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:09.001329 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-68tj5\"" Oct 30 19:19:09.001408 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:09.001387 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Oct 30 19:19:09.001557 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:09.001528 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-79b45db86b-fj8wf"] Oct 30 19:19:09.092514 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:09.092479 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0272be8c-cd91-4f7d-b183-c03a18fdadf5-plugins-config-volume\") pod \"payload-processing-external-model-79b45db86b-fj8wf\" (UID: \"0272be8c-cd91-4f7d-b183-c03a18fdadf5\") " pod="models-as-a-service/payload-processing-external-model-79b45db86b-fj8wf" Oct 30 19:19:09.092715 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:09.092693 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bv6r5\" (UniqueName: \"kubernetes.io/projected/0272be8c-cd91-4f7d-b183-c03a18fdadf5-kube-api-access-bv6r5\") pod \"payload-processing-external-model-79b45db86b-fj8wf\" (UID: \"0272be8c-cd91-4f7d-b183-c03a18fdadf5\") " pod="models-as-a-service/payload-processing-external-model-79b45db86b-fj8wf" Oct 30 19:19:09.092771 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:09.092734 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0272be8c-cd91-4f7d-b183-c03a18fdadf5-provider-credentials\") pod \"payload-processing-external-model-79b45db86b-fj8wf\" (UID: \"0272be8c-cd91-4f7d-b183-c03a18fdadf5\") " pod="models-as-a-service/payload-processing-external-model-79b45db86b-fj8wf" Oct 30 19:19:09.092855 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:09.092843 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0272be8c-cd91-4f7d-b183-c03a18fdadf5-routing-overlay-volume\") pod \"payload-processing-external-model-79b45db86b-fj8wf\" (UID: \"0272be8c-cd91-4f7d-b183-c03a18fdadf5\") " pod="models-as-a-service/payload-processing-external-model-79b45db86b-fj8wf" Oct 30 19:19:09.220168 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:09.197556 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0272be8c-cd91-4f7d-b183-c03a18fdadf5-routing-overlay-volume\") pod \"payload-processing-external-model-79b45db86b-fj8wf\" (UID: \"0272be8c-cd91-4f7d-b183-c03a18fdadf5\") " pod="models-as-a-service/payload-processing-external-model-79b45db86b-fj8wf" Oct 30 19:19:09.220168 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:09.197768 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0272be8c-cd91-4f7d-b183-c03a18fdadf5-plugins-config-volume\") pod \"payload-processing-external-model-79b45db86b-fj8wf\" (UID: \"0272be8c-cd91-4f7d-b183-c03a18fdadf5\") " pod="models-as-a-service/payload-processing-external-model-79b45db86b-fj8wf" Oct 30 19:19:09.220168 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:09.197976 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bv6r5\" (UniqueName: \"kubernetes.io/projected/0272be8c-cd91-4f7d-b183-c03a18fdadf5-kube-api-access-bv6r5\") pod \"payload-processing-external-model-79b45db86b-fj8wf\" (UID: \"0272be8c-cd91-4f7d-b183-c03a18fdadf5\") " pod="models-as-a-service/payload-processing-external-model-79b45db86b-fj8wf" Oct 30 19:19:09.220168 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:09.198020 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0272be8c-cd91-4f7d-b183-c03a18fdadf5-provider-credentials\") pod \"payload-processing-external-model-79b45db86b-fj8wf\" (UID: \"0272be8c-cd91-4f7d-b183-c03a18fdadf5\") " pod="models-as-a-service/payload-processing-external-model-79b45db86b-fj8wf" Oct 30 19:19:09.220168 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:09.208088 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0272be8c-cd91-4f7d-b183-c03a18fdadf5-plugins-config-volume\") pod \"payload-processing-external-model-79b45db86b-fj8wf\" (UID: \"0272be8c-cd91-4f7d-b183-c03a18fdadf5\") " pod="models-as-a-service/payload-processing-external-model-79b45db86b-fj8wf" Oct 30 19:19:09.220168 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:09.208694 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0272be8c-cd91-4f7d-b183-c03a18fdadf5-routing-overlay-volume\") pod \"payload-processing-external-model-79b45db86b-fj8wf\" (UID: \"0272be8c-cd91-4f7d-b183-c03a18fdadf5\") " pod="models-as-a-service/payload-processing-external-model-79b45db86b-fj8wf" Oct 30 19:19:09.220168 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:09.217743 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0272be8c-cd91-4f7d-b183-c03a18fdadf5-provider-credentials\") pod \"payload-processing-external-model-79b45db86b-fj8wf\" (UID: \"0272be8c-cd91-4f7d-b183-c03a18fdadf5\") " pod="models-as-a-service/payload-processing-external-model-79b45db86b-fj8wf" Oct 30 19:19:09.250212 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:09.249345 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bv6r5\" (UniqueName: \"kubernetes.io/projected/0272be8c-cd91-4f7d-b183-c03a18fdadf5-kube-api-access-bv6r5\") pod \"payload-processing-external-model-79b45db86b-fj8wf\" (UID: \"0272be8c-cd91-4f7d-b183-c03a18fdadf5\") " pod="models-as-a-service/payload-processing-external-model-79b45db86b-fj8wf" Oct 30 19:19:09.333171 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:09.326509 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-79b45db86b-fj8wf" Oct 30 19:19:09.652670 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:09.652354 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-79b45db86b-fj8wf"] Oct 30 19:19:09.687962 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:09.687926 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-79b45db86b-fj8wf" event={"ID":"0272be8c-cd91-4f7d-b183-c03a18fdadf5","Type":"ContainerStarted","Data":"f920b9aa17f04efd1493f5b758ab13a462205c360f4e04d6a34c83e3a5394445"} Oct 30 19:19:09.936081 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:09.935989 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:10.392690 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:10.392626 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-79b45db86b-fj8wf"] Oct 30 19:19:10.421890 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:10.421853 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:10.551665 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:10.551628 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:10.729859 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:10.728759 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-79b45db86b-fj8wf" event={"ID":"0272be8c-cd91-4f7d-b183-c03a18fdadf5","Type":"ContainerStarted","Data":"7d995d99438f6e4a54f3569497cd55f226c9e7cac68e90be96a50517afd34beb"} Oct 30 19:19:10.729859 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:10.729193 6960 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="models-as-a-service/payload-processing-external-model-79b45db86b-fj8wf" secret="" err="secret \"payload-processing-external-model-dockercfg-68tj5\" not found" Oct 30 19:19:10.730437 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:10.729865 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-79b45db86b-fj8wf" Oct 30 19:19:10.837354 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:19:10.837319 6960 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-acfdea0a628c: secret "agc-cred-acfdea0a628c" not found Oct 30 19:19:10.837354 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:19:10.837349 6960 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-79b45db86b-fj8wf: secret "agc-cred-acfdea0a628c" not found Oct 30 19:19:10.837615 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:19:10.837413 6960 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0272be8c-cd91-4f7d-b183-c03a18fdadf5-provider-credentials podName:0272be8c-cd91-4f7d-b183-c03a18fdadf5 nodeName:}" failed. No retries permitted until 2026-10-30 19:19:11.337392671 +0000 UTC m=+2718.866423805 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/0272be8c-cd91-4f7d-b183-c03a18fdadf5-provider-credentials") pod "payload-processing-external-model-79b45db86b-fj8wf" (UID: "0272be8c-cd91-4f7d-b183-c03a18fdadf5") : secret "agc-cred-acfdea0a628c" not found Oct 30 19:19:10.838112 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:19:10.838087 6960 configmap.go:193] Couldn't get configMap models-as-a-service/payload-processing-external-model-plugins: configmap "payload-processing-external-model-plugins" not found Oct 30 19:19:10.838290 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:19:10.838189 6960 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0272be8c-cd91-4f7d-b183-c03a18fdadf5-plugins-config-volume podName:0272be8c-cd91-4f7d-b183-c03a18fdadf5 nodeName:}" failed. No retries permitted until 2026-10-30 19:19:11.338134408 +0000 UTC m=+2718.867165541 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/0272be8c-cd91-4f7d-b183-c03a18fdadf5-plugins-config-volume") pod "payload-processing-external-model-79b45db86b-fj8wf" (UID: "0272be8c-cd91-4f7d-b183-c03a18fdadf5") : configmap "payload-processing-external-model-plugins" not found Oct 30 19:19:11.210622 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:11.210592 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:11.357987 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:19:11.357948 6960 configmap.go:193] Couldn't get configMap models-as-a-service/payload-processing-external-model-plugins: configmap "payload-processing-external-model-plugins" not found Oct 30 19:19:11.358279 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:19:11.358041 6960 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0272be8c-cd91-4f7d-b183-c03a18fdadf5-plugins-config-volume podName:0272be8c-cd91-4f7d-b183-c03a18fdadf5 nodeName:}" failed. No retries permitted until 2026-10-30 19:19:12.358019102 +0000 UTC m=+2719.887050237 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/0272be8c-cd91-4f7d-b183-c03a18fdadf5-plugins-config-volume") pod "payload-processing-external-model-79b45db86b-fj8wf" (UID: "0272be8c-cd91-4f7d-b183-c03a18fdadf5") : configmap "payload-processing-external-model-plugins" not found Oct 30 19:19:11.358279 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:19:11.358226 6960 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-acfdea0a628c: secret "agc-cred-acfdea0a628c" not found Oct 30 19:19:11.358279 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:19:11.358244 6960 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-79b45db86b-fj8wf: secret "agc-cred-acfdea0a628c" not found Oct 30 19:19:11.358474 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:19:11.358293 6960 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0272be8c-cd91-4f7d-b183-c03a18fdadf5-provider-credentials podName:0272be8c-cd91-4f7d-b183-c03a18fdadf5 nodeName:}" failed. No retries permitted until 2026-10-30 19:19:12.358278489 +0000 UTC m=+2719.887309606 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/0272be8c-cd91-4f7d-b183-c03a18fdadf5-provider-credentials") pod "payload-processing-external-model-79b45db86b-fj8wf" (UID: "0272be8c-cd91-4f7d-b183-c03a18fdadf5") : secret "agc-cred-acfdea0a628c" not found Oct 30 19:19:11.766257 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:11.766187 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-79b45db86b-fj8wf" podStartSLOduration=3.76613732 podStartE2EDuration="3.76613732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:19:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:19:10.772446383 +0000 UTC m=+2718.301477523" watchObservedRunningTime="2026-10-30 19:19:11.76613732 +0000 UTC m=+2719.295168454" Oct 30 19:19:12.376032 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:19:12.376003 6960 configmap.go:193] Couldn't get configMap models-as-a-service/payload-processing-external-model-plugins: configmap "payload-processing-external-model-plugins" not found Oct 30 19:19:12.376218 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:19:12.376066 6960 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0272be8c-cd91-4f7d-b183-c03a18fdadf5-plugins-config-volume podName:0272be8c-cd91-4f7d-b183-c03a18fdadf5 nodeName:}" failed. No retries permitted until 2026-10-30 19:19:14.37605185 +0000 UTC m=+2721.905082966 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/0272be8c-cd91-4f7d-b183-c03a18fdadf5-plugins-config-volume") pod "payload-processing-external-model-79b45db86b-fj8wf" (UID: "0272be8c-cd91-4f7d-b183-c03a18fdadf5") : configmap "payload-processing-external-model-plugins" not found Oct 30 19:19:12.376281 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:19:12.376245 6960 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-acfdea0a628c: secret "agc-cred-acfdea0a628c" not found Oct 30 19:19:12.376281 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:19:12.376257 6960 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-79b45db86b-fj8wf: secret "agc-cred-acfdea0a628c" not found Oct 30 19:19:12.376348 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:19:12.376296 6960 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0272be8c-cd91-4f7d-b183-c03a18fdadf5-provider-credentials podName:0272be8c-cd91-4f7d-b183-c03a18fdadf5 nodeName:}" failed. No retries permitted until 2026-10-30 19:19:14.376283962 +0000 UTC m=+2721.905315267 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/0272be8c-cd91-4f7d-b183-c03a18fdadf5-provider-credentials") pod "payload-processing-external-model-79b45db86b-fj8wf" (UID: "0272be8c-cd91-4f7d-b183-c03a18fdadf5") : secret "agc-cred-acfdea0a628c" not found Oct 30 19:19:12.685651 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:12.685551 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:13.845031 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:13.844999 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:14.412773 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:19:14.412743 6960 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-acfdea0a628c: secret "agc-cred-acfdea0a628c" not found Oct 30 19:19:14.412773 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:19:14.412767 6960 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-79b45db86b-fj8wf: secret "agc-cred-acfdea0a628c" not found Oct 30 19:19:14.413060 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:19:14.412822 6960 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0272be8c-cd91-4f7d-b183-c03a18fdadf5-provider-credentials podName:0272be8c-cd91-4f7d-b183-c03a18fdadf5 nodeName:}" failed. No retries permitted until 2026-10-30 19:19:18.412807199 +0000 UTC m=+2725.941838330 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/0272be8c-cd91-4f7d-b183-c03a18fdadf5-provider-credentials") pod "payload-processing-external-model-79b45db86b-fj8wf" (UID: "0272be8c-cd91-4f7d-b183-c03a18fdadf5") : secret "agc-cred-acfdea0a628c" not found Oct 30 19:19:14.413060 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:19:14.412821 6960 configmap.go:193] Couldn't get configMap models-as-a-service/payload-processing-external-model-plugins: configmap "payload-processing-external-model-plugins" not found Oct 30 19:19:14.413060 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:19:14.412872 6960 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0272be8c-cd91-4f7d-b183-c03a18fdadf5-plugins-config-volume podName:0272be8c-cd91-4f7d-b183-c03a18fdadf5 nodeName:}" failed. No retries permitted until 2026-10-30 19:19:18.412859556 +0000 UTC m=+2725.941890672 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/0272be8c-cd91-4f7d-b183-c03a18fdadf5-plugins-config-volume") pod "payload-processing-external-model-79b45db86b-fj8wf" (UID: "0272be8c-cd91-4f7d-b183-c03a18fdadf5") : configmap "payload-processing-external-model-plugins" not found Oct 30 19:19:14.580218 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:14.580186 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:15.593635 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:15.593603 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:16.568883 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:16.568847 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:16.737028 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:16.736993 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-79b45db86b-fj8wf" podUID="0272be8c-cd91-4f7d-b183-c03a18fdadf5" containerName="payload-processing" containerID="cri-o://7d995d99438f6e4a54f3569497cd55f226c9e7cac68e90be96a50517afd34beb" gracePeriod=25 Oct 30 19:19:17.015065 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:17.015041 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-79b45db86b-fj8wf" Oct 30 19:19:17.162265 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:17.162187 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0272be8c-cd91-4f7d-b183-c03a18fdadf5-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0272be8c-cd91-4f7d-b183-c03a18fdadf5-routing-overlay-volume\") pod \"0272be8c-cd91-4f7d-b183-c03a18fdadf5\" (UID: \"0272be8c-cd91-4f7d-b183-c03a18fdadf5\") " Oct 30 19:19:17.162446 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:17.162336 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0272be8c-cd91-4f7d-b183-c03a18fdadf5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0272be8c-cd91-4f7d-b183-c03a18fdadf5-plugins-config-volume\") pod \"0272be8c-cd91-4f7d-b183-c03a18fdadf5\" (UID: \"0272be8c-cd91-4f7d-b183-c03a18fdadf5\") " Oct 30 19:19:17.162586 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:17.162566 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0272be8c-cd91-4f7d-b183-c03a18fdadf5-kube-api-access-bv6r5\" (UniqueName: \"kubernetes.io/projected/0272be8c-cd91-4f7d-b183-c03a18fdadf5-kube-api-access-bv6r5\") pod \"0272be8c-cd91-4f7d-b183-c03a18fdadf5\" (UID: \"0272be8c-cd91-4f7d-b183-c03a18fdadf5\") " Oct 30 19:19:17.162760 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:17.162603 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0272be8c-cd91-4f7d-b183-c03a18fdadf5-routing-overlay-volume" pod "0272be8c-cd91-4f7d-b183-c03a18fdadf5" (UID: "0272be8c-cd91-4f7d-b183-c03a18fdadf5"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:19:17.162760 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:17.162627 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0272be8c-cd91-4f7d-b183-c03a18fdadf5-provider-credentials\" (UniqueName: \"kubernetes.io/projected/0272be8c-cd91-4f7d-b183-c03a18fdadf5-provider-credentials\") pod \"0272be8c-cd91-4f7d-b183-c03a18fdadf5\" (UID: \"0272be8c-cd91-4f7d-b183-c03a18fdadf5\") " Oct 30 19:19:17.162760 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:17.162712 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0272be8c-cd91-4f7d-b183-c03a18fdadf5-plugins-config-volume" pod "0272be8c-cd91-4f7d-b183-c03a18fdadf5" (UID: "0272be8c-cd91-4f7d-b183-c03a18fdadf5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:19:17.163314 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:17.163296 6960 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0272be8c-cd91-4f7d-b183-c03a18fdadf5-routing-overlay-volume\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:19:17.163417 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:17.163319 6960 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0272be8c-cd91-4f7d-b183-c03a18fdadf5-plugins-config-volume\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:19:17.165015 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:17.164987 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0272be8c-cd91-4f7d-b183-c03a18fdadf5-kube-api-access-bv6r5" pod "0272be8c-cd91-4f7d-b183-c03a18fdadf5" (UID: "0272be8c-cd91-4f7d-b183-c03a18fdadf5"). InnerVolumeSpecName "kube-api-access-bv6r5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:19:17.165101 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:17.165039 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0272be8c-cd91-4f7d-b183-c03a18fdadf5-provider-credentials" pod "0272be8c-cd91-4f7d-b183-c03a18fdadf5" (UID: "0272be8c-cd91-4f7d-b183-c03a18fdadf5"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:19:17.264670 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:17.264639 6960 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0272be8c-cd91-4f7d-b183-c03a18fdadf5-provider-credentials\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:19:17.264670 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:17.264668 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bv6r5\" (UniqueName: \"kubernetes.io/projected/0272be8c-cd91-4f7d-b183-c03a18fdadf5-kube-api-access-bv6r5\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:19:17.564227 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:17.564129 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:17.787700 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:17.787659 6960 generic.go:356] "Generic (PLEG): container finished" podID="0272be8c-cd91-4f7d-b183-c03a18fdadf5" containerID="7d995d99438f6e4a54f3569497cd55f226c9e7cac68e90be96a50517afd34beb" exitCode=0 Oct 30 19:19:17.788178 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:17.787709 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-79b45db86b-fj8wf" Oct 30 19:19:17.788178 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:17.787745 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-79b45db86b-fj8wf" event={"ID":"0272be8c-cd91-4f7d-b183-c03a18fdadf5","Type":"ContainerDied","Data":"7d995d99438f6e4a54f3569497cd55f226c9e7cac68e90be96a50517afd34beb"} Oct 30 19:19:17.788178 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:17.787799 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-79b45db86b-fj8wf" event={"ID":"0272be8c-cd91-4f7d-b183-c03a18fdadf5","Type":"ContainerDied","Data":"f920b9aa17f04efd1493f5b758ab13a462205c360f4e04d6a34c83e3a5394445"} Oct 30 19:19:17.788178 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:17.787823 6960 scope.go:122] "RemoveContainer" containerID="7d995d99438f6e4a54f3569497cd55f226c9e7cac68e90be96a50517afd34beb" Oct 30 19:19:17.799538 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:17.799521 6960 scope.go:122] "RemoveContainer" containerID="7d995d99438f6e4a54f3569497cd55f226c9e7cac68e90be96a50517afd34beb" Oct 30 19:19:17.799791 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:19:17.799768 6960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d995d99438f6e4a54f3569497cd55f226c9e7cac68e90be96a50517afd34beb\": container with ID starting with 7d995d99438f6e4a54f3569497cd55f226c9e7cac68e90be96a50517afd34beb not found: ID does not exist" containerID="7d995d99438f6e4a54f3569497cd55f226c9e7cac68e90be96a50517afd34beb" Oct 30 19:19:17.799841 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:17.799799 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d995d99438f6e4a54f3569497cd55f226c9e7cac68e90be96a50517afd34beb"} err="failed to get container status \"7d995d99438f6e4a54f3569497cd55f226c9e7cac68e90be96a50517afd34beb\": rpc error: code = NotFound desc = could not find container \"7d995d99438f6e4a54f3569497cd55f226c9e7cac68e90be96a50517afd34beb\": container with ID starting with 7d995d99438f6e4a54f3569497cd55f226c9e7cac68e90be96a50517afd34beb not found: ID does not exist" Oct 30 19:19:17.817260 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:17.817205 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-79b45db86b-fj8wf"] Oct 30 19:19:17.821798 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:17.821778 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-79b45db86b-fj8wf"] Oct 30 19:19:18.415477 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:18.415429 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:18.595784 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:18.595754 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0272be8c-cd91-4f7d-b183-c03a18fdadf5" path="/var/lib/kubelet/pods/0272be8c-cd91-4f7d-b183-c03a18fdadf5/volumes" Oct 30 19:19:19.231447 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:19.231407 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:19.985207 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:19.985170 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-689c23-58c9d7c564-ncqml"] Oct 30 19:19:19.985406 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:19.985385 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-689c23-58c9d7c564-ncqml" podUID="0ac809bd-4083-4519-b7b5-903a22885985" containerName="maas-api" containerID="cri-o://6e9eb76380dd6c618dff8789cc7a5974398567bb6326177c3fe33528745450d5" gracePeriod=30 Oct 30 19:19:20.088949 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:20.088915 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:20.448169 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:20.447786 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:20.567671 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:20.567646 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-689c23-58c9d7c564-ncqml" Oct 30 19:19:20.640425 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:20.640388 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:20.750482 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:20.750384 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ac809bd-4083-4519-b7b5-903a22885985-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ac809bd-4083-4519-b7b5-903a22885985-metrics-certs\") pod \"0ac809bd-4083-4519-b7b5-903a22885985\" (UID: \"0ac809bd-4083-4519-b7b5-903a22885985\") " Oct 30 19:19:20.750666 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:20.750595 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ac809bd-4083-4519-b7b5-903a22885985-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ac809bd-4083-4519-b7b5-903a22885985-openshift-service-ca\") pod \"0ac809bd-4083-4519-b7b5-903a22885985\" (UID: \"0ac809bd-4083-4519-b7b5-903a22885985\") " Oct 30 19:19:20.750666 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:20.750634 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ac809bd-4083-4519-b7b5-903a22885985-kube-api-access-kmrt4\" (UniqueName: \"kubernetes.io/projected/0ac809bd-4083-4519-b7b5-903a22885985-kube-api-access-kmrt4\") pod \"0ac809bd-4083-4519-b7b5-903a22885985\" (UID: \"0ac809bd-4083-4519-b7b5-903a22885985\") " Oct 30 19:19:20.750799 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:20.750673 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ac809bd-4083-4519-b7b5-903a22885985-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ac809bd-4083-4519-b7b5-903a22885985-maas-api-tls\") pod \"0ac809bd-4083-4519-b7b5-903a22885985\" (UID: \"0ac809bd-4083-4519-b7b5-903a22885985\") " Oct 30 19:19:20.751042 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:20.751012 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ac809bd-4083-4519-b7b5-903a22885985-openshift-service-ca" pod "0ac809bd-4083-4519-b7b5-903a22885985" (UID: "0ac809bd-4083-4519-b7b5-903a22885985"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:19:20.753071 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:20.752482 6960 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ac809bd-4083-4519-b7b5-903a22885985-openshift-service-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:19:20.753720 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:20.753690 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ac809bd-4083-4519-b7b5-903a22885985-kube-api-access-kmrt4" pod "0ac809bd-4083-4519-b7b5-903a22885985" (UID: "0ac809bd-4083-4519-b7b5-903a22885985"). InnerVolumeSpecName "kube-api-access-kmrt4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:19:20.753848 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:20.753813 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ac809bd-4083-4519-b7b5-903a22885985-maas-api-tls" pod "0ac809bd-4083-4519-b7b5-903a22885985" (UID: "0ac809bd-4083-4519-b7b5-903a22885985"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:19:20.754733 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:20.754669 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ac809bd-4083-4519-b7b5-903a22885985-metrics-certs" pod "0ac809bd-4083-4519-b7b5-903a22885985" (UID: "0ac809bd-4083-4519-b7b5-903a22885985"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:19:20.787026 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:20.786935 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:20.818696 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:20.818666 6960 generic.go:356] "Generic (PLEG): container finished" podID="0ac809bd-4083-4519-b7b5-903a22885985" containerID="6e9eb76380dd6c618dff8789cc7a5974398567bb6326177c3fe33528745450d5" exitCode=0 Oct 30 19:19:20.818888 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:20.818788 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-689c23-58c9d7c564-ncqml" Oct 30 19:19:20.818888 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:20.818812 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-689c23-58c9d7c564-ncqml" event={"ID":"0ac809bd-4083-4519-b7b5-903a22885985","Type":"ContainerDied","Data":"6e9eb76380dd6c618dff8789cc7a5974398567bb6326177c3fe33528745450d5"} Oct 30 19:19:20.818888 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:20.818876 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-689c23-58c9d7c564-ncqml" event={"ID":"0ac809bd-4083-4519-b7b5-903a22885985","Type":"ContainerDied","Data":"a18bee4d014c8dc17cc21ef0809c5c912d4331855b90b56bad924a340d7467ae"} Oct 30 19:19:20.819129 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:20.818904 6960 scope.go:122] "RemoveContainer" containerID="6e9eb76380dd6c618dff8789cc7a5974398567bb6326177c3fe33528745450d5" Oct 30 19:19:20.831559 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:20.831543 6960 scope.go:122] "RemoveContainer" containerID="6e9eb76380dd6c618dff8789cc7a5974398567bb6326177c3fe33528745450d5" Oct 30 19:19:20.831829 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:19:20.831811 6960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e9eb76380dd6c618dff8789cc7a5974398567bb6326177c3fe33528745450d5\": container with ID starting with 6e9eb76380dd6c618dff8789cc7a5974398567bb6326177c3fe33528745450d5 not found: ID does not exist" containerID="6e9eb76380dd6c618dff8789cc7a5974398567bb6326177c3fe33528745450d5" Oct 30 19:19:20.831905 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:20.831837 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e9eb76380dd6c618dff8789cc7a5974398567bb6326177c3fe33528745450d5"} err="failed to get container status \"6e9eb76380dd6c618dff8789cc7a5974398567bb6326177c3fe33528745450d5\": rpc error: code = NotFound desc = could not find container \"6e9eb76380dd6c618dff8789cc7a5974398567bb6326177c3fe33528745450d5\": container with ID starting with 6e9eb76380dd6c618dff8789cc7a5974398567bb6326177c3fe33528745450d5 not found: ID does not exist" Oct 30 19:19:20.853973 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:20.853949 6960 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ac809bd-4083-4519-b7b5-903a22885985-metrics-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:19:20.853973 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:20.853971 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmrt4\" (UniqueName: \"kubernetes.io/projected/0ac809bd-4083-4519-b7b5-903a22885985-kube-api-access-kmrt4\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:19:20.854120 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:20.853981 6960 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ac809bd-4083-4519-b7b5-903a22885985-maas-api-tls\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:19:20.860318 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:20.860293 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-689c23-58c9d7c564-ncqml"] Oct 30 19:19:20.864307 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:20.864286 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-689c23-58c9d7c564-ncqml"] Oct 30 19:19:21.355575 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:21.355541 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:21.866222 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:21.866183 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:22.238169 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:22.238068 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:22.469168 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:22.468702 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:22.641312 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:22.641274 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ac809bd-4083-4519-b7b5-903a22885985" path="/var/lib/kubelet/pods/0ac809bd-4083-4519-b7b5-903a22885985/volumes" Oct 30 19:19:23.239474 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:23.239440 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:23.934320 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:19:23.934283 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 19:19:24.257461 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:24.251657 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:25.132086 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:25.132047 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:26.119826 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:26.119795 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:26.835366 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:26.835162 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:27.001208 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:26.985636 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:27.864322 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:27.864285 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:28.817958 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:28.817917 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:29.370885 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:29.370853 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:29.504420 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:29.504388 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:29.805608 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:29.805583 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-5f7lv"] Oct 30 19:19:29.805811 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:29.805783 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-5f7lv" podUID="755750a6-9b02-43c7-adfd-00f2073020fe" containerName="istio-proxy" containerID="cri-o://f6a1724d2ff162b697d3d8fd0bd2737e9383a6b193a237e7539db8eb400a6902" gracePeriod=30 Oct 30 19:19:29.891333 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:29.891291 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:30.055355 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:30.055301 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:31.087052 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:31.087016 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:31.669715 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:31.669680 6960 patch_prober.go:29] interesting pod/e2e-extmod-a-689c23-openshift-default-57dff78645-5f7lv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:19:31.669945 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:31.669749 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-5f7lv" podUID="755750a6-9b02-43c7-adfd-00f2073020fe" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:19:31.899476 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:31.899445 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:32.848191 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:32.848138 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:33.825868 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:33.825832 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:34.829667 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:34.829632 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:35.064571 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:35.064379 6960 generic.go:356] "Generic (PLEG): container finished" podID="755750a6-9b02-43c7-adfd-00f2073020fe" containerID="f6a1724d2ff162b697d3d8fd0bd2737e9383a6b193a237e7539db8eb400a6902" exitCode=0 Oct 30 19:19:35.064571 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:35.064500 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-5f7lv" event={"ID":"755750a6-9b02-43c7-adfd-00f2073020fe","Type":"ContainerDied","Data":"f6a1724d2ff162b697d3d8fd0bd2737e9383a6b193a237e7539db8eb400a6902"} Oct 30 19:19:35.361348 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:35.359910 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-5f7lv" Oct 30 19:19:35.542684 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:35.542597 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/755750a6-9b02-43c7-adfd-00f2073020fe-kube-api-access-gdgvq\" (UniqueName: \"kubernetes.io/projected/755750a6-9b02-43c7-adfd-00f2073020fe-kube-api-access-gdgvq\") pod \"755750a6-9b02-43c7-adfd-00f2073020fe\" (UID: \"755750a6-9b02-43c7-adfd-00f2073020fe\") " Oct 30 19:19:35.542684 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:35.542658 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/755750a6-9b02-43c7-adfd-00f2073020fe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/755750a6-9b02-43c7-adfd-00f2073020fe-workload-certs\") pod \"755750a6-9b02-43c7-adfd-00f2073020fe\" (UID: \"755750a6-9b02-43c7-adfd-00f2073020fe\") " Oct 30 19:19:35.542879 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:35.542801 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/755750a6-9b02-43c7-adfd-00f2073020fe-istio-token\" (UniqueName: \"kubernetes.io/projected/755750a6-9b02-43c7-adfd-00f2073020fe-istio-token\") pod \"755750a6-9b02-43c7-adfd-00f2073020fe\" (UID: \"755750a6-9b02-43c7-adfd-00f2073020fe\") " Oct 30 19:19:35.542879 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:35.542833 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/755750a6-9b02-43c7-adfd-00f2073020fe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/755750a6-9b02-43c7-adfd-00f2073020fe-istio-podinfo\") pod \"755750a6-9b02-43c7-adfd-00f2073020fe\" (UID: \"755750a6-9b02-43c7-adfd-00f2073020fe\") " Oct 30 19:19:35.542954 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:35.542874 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/755750a6-9b02-43c7-adfd-00f2073020fe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/755750a6-9b02-43c7-adfd-00f2073020fe-workload-socket\") pod \"755750a6-9b02-43c7-adfd-00f2073020fe\" (UID: \"755750a6-9b02-43c7-adfd-00f2073020fe\") " Oct 30 19:19:35.543035 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:35.543020 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/755750a6-9b02-43c7-adfd-00f2073020fe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/755750a6-9b02-43c7-adfd-00f2073020fe-istio-data\") pod \"755750a6-9b02-43c7-adfd-00f2073020fe\" (UID: \"755750a6-9b02-43c7-adfd-00f2073020fe\") " Oct 30 19:19:35.543092 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:35.543074 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/755750a6-9b02-43c7-adfd-00f2073020fe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/755750a6-9b02-43c7-adfd-00f2073020fe-credential-socket\") pod \"755750a6-9b02-43c7-adfd-00f2073020fe\" (UID: \"755750a6-9b02-43c7-adfd-00f2073020fe\") " Oct 30 19:19:35.543188 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:35.543168 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/755750a6-9b02-43c7-adfd-00f2073020fe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/755750a6-9b02-43c7-adfd-00f2073020fe-istio-envoy\") pod \"755750a6-9b02-43c7-adfd-00f2073020fe\" (UID: \"755750a6-9b02-43c7-adfd-00f2073020fe\") " Oct 30 19:19:35.543314 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:35.543297 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/755750a6-9b02-43c7-adfd-00f2073020fe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/755750a6-9b02-43c7-adfd-00f2073020fe-istiod-ca-cert\") pod \"755750a6-9b02-43c7-adfd-00f2073020fe\" (UID: \"755750a6-9b02-43c7-adfd-00f2073020fe\") " Oct 30 19:19:35.544069 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:35.543934 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/755750a6-9b02-43c7-adfd-00f2073020fe-istio-data" pod "755750a6-9b02-43c7-adfd-00f2073020fe" (UID: "755750a6-9b02-43c7-adfd-00f2073020fe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:19:35.544485 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:35.544243 6960 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/755750a6-9b02-43c7-adfd-00f2073020fe-istio-data\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:19:35.544485 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:35.544271 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/755750a6-9b02-43c7-adfd-00f2073020fe-workload-socket" pod "755750a6-9b02-43c7-adfd-00f2073020fe" (UID: "755750a6-9b02-43c7-adfd-00f2073020fe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:19:35.544485 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:35.544316 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/755750a6-9b02-43c7-adfd-00f2073020fe-istiod-ca-cert" pod "755750a6-9b02-43c7-adfd-00f2073020fe" (UID: "755750a6-9b02-43c7-adfd-00f2073020fe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:19:35.544485 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:35.544359 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/755750a6-9b02-43c7-adfd-00f2073020fe-credential-socket" pod "755750a6-9b02-43c7-adfd-00f2073020fe" (UID: "755750a6-9b02-43c7-adfd-00f2073020fe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:19:35.544485 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:35.544413 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/755750a6-9b02-43c7-adfd-00f2073020fe-workload-certs" pod "755750a6-9b02-43c7-adfd-00f2073020fe" (UID: "755750a6-9b02-43c7-adfd-00f2073020fe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:19:35.546211 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:35.546121 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/755750a6-9b02-43c7-adfd-00f2073020fe-istio-podinfo" pod "755750a6-9b02-43c7-adfd-00f2073020fe" (UID: "755750a6-9b02-43c7-adfd-00f2073020fe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:19:35.546663 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:35.546631 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/755750a6-9b02-43c7-adfd-00f2073020fe-istio-token" pod "755750a6-9b02-43c7-adfd-00f2073020fe" (UID: "755750a6-9b02-43c7-adfd-00f2073020fe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:19:35.546887 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:35.546859 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/755750a6-9b02-43c7-adfd-00f2073020fe-kube-api-access-gdgvq" pod "755750a6-9b02-43c7-adfd-00f2073020fe" (UID: "755750a6-9b02-43c7-adfd-00f2073020fe"). InnerVolumeSpecName "kube-api-access-gdgvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:19:35.547396 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:35.547366 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/755750a6-9b02-43c7-adfd-00f2073020fe-istio-envoy" pod "755750a6-9b02-43c7-adfd-00f2073020fe" (UID: "755750a6-9b02-43c7-adfd-00f2073020fe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:19:35.645131 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:35.645099 6960 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/755750a6-9b02-43c7-adfd-00f2073020fe-istiod-ca-cert\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:19:35.645131 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:35.645127 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdgvq\" (UniqueName: \"kubernetes.io/projected/755750a6-9b02-43c7-adfd-00f2073020fe-kube-api-access-gdgvq\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:19:35.645327 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:35.645154 6960 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/755750a6-9b02-43c7-adfd-00f2073020fe-workload-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:19:35.645327 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:35.645164 6960 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/755750a6-9b02-43c7-adfd-00f2073020fe-istio-token\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:19:35.645327 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:35.645174 6960 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/755750a6-9b02-43c7-adfd-00f2073020fe-istio-podinfo\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:19:35.645327 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:35.645183 6960 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/755750a6-9b02-43c7-adfd-00f2073020fe-workload-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:19:35.645327 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:35.645191 6960 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/755750a6-9b02-43c7-adfd-00f2073020fe-credential-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:19:35.645327 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:35.645200 6960 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/755750a6-9b02-43c7-adfd-00f2073020fe-istio-envoy\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:19:36.086172 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:36.082666 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-5f7lv" event={"ID":"755750a6-9b02-43c7-adfd-00f2073020fe","Type":"ContainerDied","Data":"1feec519a939586556f8852217cc758fe9317bebbe6247d7cf21cdac9c408382"} Oct 30 19:19:36.086172 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:36.082741 6960 scope.go:122] "RemoveContainer" containerID="f6a1724d2ff162b697d3d8fd0bd2737e9383a6b193a237e7539db8eb400a6902" Oct 30 19:19:36.086172 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:36.082894 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-5f7lv" Oct 30 19:19:36.124435 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:36.124407 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-5f7lv"] Oct 30 19:19:36.129877 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:36.129854 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-689c23-openshift-default-57dff78645-5f7lv"] Oct 30 19:19:36.256291 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:36.256254 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:36.599165 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:36.599115 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="755750a6-9b02-43c7-adfd-00f2073020fe" path="/var/lib/kubelet/pods/755750a6-9b02-43c7-adfd-00f2073020fe/volumes" Oct 30 19:19:36.743276 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:36.743245 6960 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-zldp5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:19:36.743441 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:36.743329 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zldp5" podUID="d12464a9-d9c7-46e6-b3cc-ab0c16d75070" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:19:37.070234 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:37.070202 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:37.977977 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:37.977873 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:38.894775 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:38.894747 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:39.845436 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:39.845406 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:40.542748 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:40.542712 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:40.660642 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:40.660612 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:40.772562 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:40.772523 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:41.618760 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:41.618726 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:42.732717 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:42.732680 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:43.706951 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:43.706918 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:44.532193 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:44.532158 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:45.322983 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:45.322949 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:46.031663 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:46.031631 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-e8c554-6dcfccfdb5-wcdsr"] Oct 30 19:19:46.032430 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:46.032397 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e8c554-6dcfccfdb5-wcdsr" podUID="3e8ff0f2-eaf3-434d-b877-f95964be8923" containerName="maas-api" containerID="cri-o://553e0dc757254c4c3b47f581bc0972626a9af81c65d46adc6091b7bcada337c6" gracePeriod=30 Oct 30 19:19:46.110845 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:46.110813 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:46.186688 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:46.186640 6960 generic.go:356] "Generic (PLEG): container finished" podID="3e8ff0f2-eaf3-434d-b877-f95964be8923" containerID="553e0dc757254c4c3b47f581bc0972626a9af81c65d46adc6091b7bcada337c6" exitCode=0 Oct 30 19:19:46.186872 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:46.186720 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e8c554-6dcfccfdb5-wcdsr" event={"ID":"3e8ff0f2-eaf3-434d-b877-f95964be8923","Type":"ContainerDied","Data":"553e0dc757254c4c3b47f581bc0972626a9af81c65d46adc6091b7bcada337c6"} Oct 30 19:19:46.463512 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:46.463420 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:46.567548 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:46.567522 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e8c554-6dcfccfdb5-wcdsr" Oct 30 19:19:46.603683 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:46.598587 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:46.724920 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:46.724830 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e8ff0f2-eaf3-434d-b877-f95964be8923-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e8ff0f2-eaf3-434d-b877-f95964be8923-openshift-service-ca\") pod \"3e8ff0f2-eaf3-434d-b877-f95964be8923\" (UID: \"3e8ff0f2-eaf3-434d-b877-f95964be8923\") " Oct 30 19:19:46.725099 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:46.724961 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e8ff0f2-eaf3-434d-b877-f95964be8923-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e8ff0f2-eaf3-434d-b877-f95964be8923-metrics-certs\") pod \"3e8ff0f2-eaf3-434d-b877-f95964be8923\" (UID: \"3e8ff0f2-eaf3-434d-b877-f95964be8923\") " Oct 30 19:19:46.725099 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:46.724997 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e8ff0f2-eaf3-434d-b877-f95964be8923-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e8ff0f2-eaf3-434d-b877-f95964be8923-maas-api-tls\") pod \"3e8ff0f2-eaf3-434d-b877-f95964be8923\" (UID: \"3e8ff0f2-eaf3-434d-b877-f95964be8923\") " Oct 30 19:19:46.725099 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:46.725076 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e8ff0f2-eaf3-434d-b877-f95964be8923-kube-api-access-p8kfr\" (UniqueName: \"kubernetes.io/projected/3e8ff0f2-eaf3-434d-b877-f95964be8923-kube-api-access-p8kfr\") pod \"3e8ff0f2-eaf3-434d-b877-f95964be8923\" (UID: \"3e8ff0f2-eaf3-434d-b877-f95964be8923\") " Oct 30 19:19:46.725318 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:46.725230 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e8ff0f2-eaf3-434d-b877-f95964be8923-openshift-service-ca" pod "3e8ff0f2-eaf3-434d-b877-f95964be8923" (UID: "3e8ff0f2-eaf3-434d-b877-f95964be8923"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:19:46.725785 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:46.725762 6960 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e8ff0f2-eaf3-434d-b877-f95964be8923-openshift-service-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:19:46.727422 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:46.727402 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e8ff0f2-eaf3-434d-b877-f95964be8923-maas-api-tls" pod "3e8ff0f2-eaf3-434d-b877-f95964be8923" (UID: "3e8ff0f2-eaf3-434d-b877-f95964be8923"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:19:46.727506 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:46.727418 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e8ff0f2-eaf3-434d-b877-f95964be8923-metrics-certs" pod "3e8ff0f2-eaf3-434d-b877-f95964be8923" (UID: "3e8ff0f2-eaf3-434d-b877-f95964be8923"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:19:46.727567 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:46.727550 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e8ff0f2-eaf3-434d-b877-f95964be8923-kube-api-access-p8kfr" pod "3e8ff0f2-eaf3-434d-b877-f95964be8923" (UID: "3e8ff0f2-eaf3-434d-b877-f95964be8923"). InnerVolumeSpecName "kube-api-access-p8kfr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:19:46.753467 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:46.753435 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:46.826998 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:46.826968 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8kfr\" (UniqueName: \"kubernetes.io/projected/3e8ff0f2-eaf3-434d-b877-f95964be8923-kube-api-access-p8kfr\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:19:46.826998 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:46.826995 6960 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e8ff0f2-eaf3-434d-b877-f95964be8923-metrics-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:19:46.826998 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:46.827005 6960 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e8ff0f2-eaf3-434d-b877-f95964be8923-maas-api-tls\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:19:47.075627 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:47.075598 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:47.196216 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:47.196189 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e8c554-6dcfccfdb5-wcdsr" Oct 30 19:19:47.196216 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:47.196203 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e8c554-6dcfccfdb5-wcdsr" event={"ID":"3e8ff0f2-eaf3-434d-b877-f95964be8923","Type":"ContainerDied","Data":"70dd633a4adfbcfc39f6f44ad2d153fddee66f59cb5f70475811b6f714aa15e0"} Oct 30 19:19:47.196431 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:47.196273 6960 scope.go:122] "RemoveContainer" containerID="553e0dc757254c4c3b47f581bc0972626a9af81c65d46adc6091b7bcada337c6" Oct 30 19:19:47.231015 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:47.230990 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-e8c554-6dcfccfdb5-wcdsr"] Oct 30 19:19:47.281507 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:47.281479 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-e8c554-6dcfccfdb5-wcdsr"] Oct 30 19:19:47.832638 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:47.832594 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:48.630136 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:48.630096 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e8ff0f2-eaf3-434d-b877-f95964be8923" path="/var/lib/kubelet/pods/3e8ff0f2-eaf3-434d-b877-f95964be8923/volumes" Oct 30 19:19:48.763929 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:48.763893 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:49.758416 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:49.758377 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:49.871828 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:49.871765 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:50.320237 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:50.320198 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:50.442376 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:50.442344 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:50.900562 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:50.900524 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:52.105406 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:52.105372 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:52.864224 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:52.864191 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:53.418742 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:53.418707 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-wfw84"] Oct 30 19:19:53.419188 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:53.418921 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-wfw84" podUID="da02a9d7-7b75-401c-9134-8e45d7303175" containerName="istio-proxy" containerID="cri-o://30633a687fadded7a1baf9dd9d6a60ebd1a07128924ad553e4a1430a0ccfc5c1" gracePeriod=30 Oct 30 19:19:53.481031 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:53.480991 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-e77b1d-6c5d97c974-5r5f8"] Oct 30 19:19:53.481345 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:53.481316 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e77b1d-6c5d97c974-5r5f8" podUID="56a9fce0-2511-4217-afb8-feb2184fb974" containerName="maas-api" containerID="cri-o://a4528dde609c55db773c04d61851f2d6c6d8c43ba3112fd0e92f476621acf89c" gracePeriod=30 Oct 30 19:19:53.565337 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:53.565302 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:53.963293 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:53.960489 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:54.056384 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:19:54.056348 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 19:19:54.080550 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:54.080520 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:54.084294 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:54.084273 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e77b1d-6c5d97c974-5r5f8" Oct 30 19:19:54.208530 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:54.208489 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:54.247543 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:54.244418 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56a9fce0-2511-4217-afb8-feb2184fb974-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56a9fce0-2511-4217-afb8-feb2184fb974-openshift-service-ca\") pod \"56a9fce0-2511-4217-afb8-feb2184fb974\" (UID: \"56a9fce0-2511-4217-afb8-feb2184fb974\") " Oct 30 19:19:54.247543 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:54.244564 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56a9fce0-2511-4217-afb8-feb2184fb974-metrics-certs\" (UniqueName: \"kubernetes.io/secret/56a9fce0-2511-4217-afb8-feb2184fb974-metrics-certs\") pod \"56a9fce0-2511-4217-afb8-feb2184fb974\" (UID: \"56a9fce0-2511-4217-afb8-feb2184fb974\") " Oct 30 19:19:54.247543 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:54.244648 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56a9fce0-2511-4217-afb8-feb2184fb974-kube-api-access-kft5m\" (UniqueName: \"kubernetes.io/projected/56a9fce0-2511-4217-afb8-feb2184fb974-kube-api-access-kft5m\") pod \"56a9fce0-2511-4217-afb8-feb2184fb974\" (UID: \"56a9fce0-2511-4217-afb8-feb2184fb974\") " Oct 30 19:19:54.247543 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:54.245937 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56a9fce0-2511-4217-afb8-feb2184fb974-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56a9fce0-2511-4217-afb8-feb2184fb974-maas-api-tls\") pod \"56a9fce0-2511-4217-afb8-feb2184fb974\" (UID: \"56a9fce0-2511-4217-afb8-feb2184fb974\") " Oct 30 19:19:54.247543 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:54.246213 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56a9fce0-2511-4217-afb8-feb2184fb974-openshift-service-ca" pod "56a9fce0-2511-4217-afb8-feb2184fb974" (UID: "56a9fce0-2511-4217-afb8-feb2184fb974"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:19:54.250116 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:54.249430 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56a9fce0-2511-4217-afb8-feb2184fb974-maas-api-tls" pod "56a9fce0-2511-4217-afb8-feb2184fb974" (UID: "56a9fce0-2511-4217-afb8-feb2184fb974"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:19:54.250519 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:54.250491 6960 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56a9fce0-2511-4217-afb8-feb2184fb974-maas-api-tls\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:19:54.250739 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:54.250522 6960 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56a9fce0-2511-4217-afb8-feb2184fb974-openshift-service-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:19:54.256041 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:54.253702 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56a9fce0-2511-4217-afb8-feb2184fb974-kube-api-access-kft5m" pod "56a9fce0-2511-4217-afb8-feb2184fb974" (UID: "56a9fce0-2511-4217-afb8-feb2184fb974"). InnerVolumeSpecName "kube-api-access-kft5m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:19:54.256041 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:54.255858 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56a9fce0-2511-4217-afb8-feb2184fb974-metrics-certs" pod "56a9fce0-2511-4217-afb8-feb2184fb974" (UID: "56a9fce0-2511-4217-afb8-feb2184fb974"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:19:54.274284 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:54.274254 6960 generic.go:356] "Generic (PLEG): container finished" podID="56a9fce0-2511-4217-afb8-feb2184fb974" containerID="a4528dde609c55db773c04d61851f2d6c6d8c43ba3112fd0e92f476621acf89c" exitCode=0 Oct 30 19:19:54.274446 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:54.274305 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e77b1d-6c5d97c974-5r5f8" Oct 30 19:19:54.274446 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:54.274334 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e77b1d-6c5d97c974-5r5f8" event={"ID":"56a9fce0-2511-4217-afb8-feb2184fb974","Type":"ContainerDied","Data":"a4528dde609c55db773c04d61851f2d6c6d8c43ba3112fd0e92f476621acf89c"} Oct 30 19:19:54.274446 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:54.274390 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e77b1d-6c5d97c974-5r5f8" event={"ID":"56a9fce0-2511-4217-afb8-feb2184fb974","Type":"ContainerDied","Data":"2095306065e3c8931b0a70a30af6eedf917fe22f43b532e0f6c1c6f2da61d041"} Oct 30 19:19:54.274446 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:54.274414 6960 scope.go:122] "RemoveContainer" containerID="a4528dde609c55db773c04d61851f2d6c6d8c43ba3112fd0e92f476621acf89c" Oct 30 19:19:54.297473 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:54.297448 6960 scope.go:122] "RemoveContainer" containerID="a4528dde609c55db773c04d61851f2d6c6d8c43ba3112fd0e92f476621acf89c" Oct 30 19:19:54.297827 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:19:54.297802 6960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4528dde609c55db773c04d61851f2d6c6d8c43ba3112fd0e92f476621acf89c\": container with ID starting with a4528dde609c55db773c04d61851f2d6c6d8c43ba3112fd0e92f476621acf89c not found: ID does not exist" containerID="a4528dde609c55db773c04d61851f2d6c6d8c43ba3112fd0e92f476621acf89c" Oct 30 19:19:54.297893 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:54.297842 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4528dde609c55db773c04d61851f2d6c6d8c43ba3112fd0e92f476621acf89c"} err="failed to get container status \"a4528dde609c55db773c04d61851f2d6c6d8c43ba3112fd0e92f476621acf89c\": rpc error: code = NotFound desc = could not find container \"a4528dde609c55db773c04d61851f2d6c6d8c43ba3112fd0e92f476621acf89c\": container with ID starting with a4528dde609c55db773c04d61851f2d6c6d8c43ba3112fd0e92f476621acf89c not found: ID does not exist" Oct 30 19:19:54.308267 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:54.308242 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-e77b1d-6c5d97c974-5r5f8"] Oct 30 19:19:54.314898 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:54.314879 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-e77b1d-6c5d97c974-5r5f8"] Oct 30 19:19:54.351975 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:54.351948 6960 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56a9fce0-2511-4217-afb8-feb2184fb974-metrics-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:19:54.351975 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:54.351978 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kft5m\" (UniqueName: \"kubernetes.io/projected/56a9fce0-2511-4217-afb8-feb2184fb974-kube-api-access-kft5m\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:19:54.445646 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:54.445614 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:54.569164 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:54.569011 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:54.595713 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:54.595682 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56a9fce0-2511-4217-afb8-feb2184fb974" path="/var/lib/kubelet/pods/56a9fce0-2511-4217-afb8-feb2184fb974/volumes" Oct 30 19:19:54.907393 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:54.907311 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:55.029700 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:55.029664 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:55.222882 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:55.222795 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:55.472168 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:55.471673 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:56.323905 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:56.323865 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:57.220624 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:57.220590 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:58.002751 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:58.002719 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:58.128239 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:58.128138 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:58.722520 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:58.722483 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-wfw84" Oct 30 19:19:58.737393 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:58.737359 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da02a9d7-7b75-401c-9134-8e45d7303175-istio-token\" (UniqueName: \"kubernetes.io/projected/da02a9d7-7b75-401c-9134-8e45d7303175-istio-token\") pod \"da02a9d7-7b75-401c-9134-8e45d7303175\" (UID: \"da02a9d7-7b75-401c-9134-8e45d7303175\") " Oct 30 19:19:58.737566 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:58.737548 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da02a9d7-7b75-401c-9134-8e45d7303175-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da02a9d7-7b75-401c-9134-8e45d7303175-istiod-ca-cert\") pod \"da02a9d7-7b75-401c-9134-8e45d7303175\" (UID: \"da02a9d7-7b75-401c-9134-8e45d7303175\") " Oct 30 19:19:58.737660 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:58.737645 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/da02a9d7-7b75-401c-9134-8e45d7303175-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da02a9d7-7b75-401c-9134-8e45d7303175-istio-podinfo\") pod \"da02a9d7-7b75-401c-9134-8e45d7303175\" (UID: \"da02a9d7-7b75-401c-9134-8e45d7303175\") " Oct 30 19:19:58.737713 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:58.737702 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da02a9d7-7b75-401c-9134-8e45d7303175-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da02a9d7-7b75-401c-9134-8e45d7303175-workload-certs\") pod \"da02a9d7-7b75-401c-9134-8e45d7303175\" (UID: \"da02a9d7-7b75-401c-9134-8e45d7303175\") " Oct 30 19:19:58.737770 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:58.737732 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da02a9d7-7b75-401c-9134-8e45d7303175-kube-api-access-85wvp\" (UniqueName: \"kubernetes.io/projected/da02a9d7-7b75-401c-9134-8e45d7303175-kube-api-access-85wvp\") pod \"da02a9d7-7b75-401c-9134-8e45d7303175\" (UID: \"da02a9d7-7b75-401c-9134-8e45d7303175\") " Oct 30 19:19:58.737854 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:58.737832 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da02a9d7-7b75-401c-9134-8e45d7303175-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da02a9d7-7b75-401c-9134-8e45d7303175-credential-socket\") pod \"da02a9d7-7b75-401c-9134-8e45d7303175\" (UID: \"da02a9d7-7b75-401c-9134-8e45d7303175\") " Oct 30 19:19:58.738064 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:58.737917 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da02a9d7-7b75-401c-9134-8e45d7303175-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da02a9d7-7b75-401c-9134-8e45d7303175-istio-envoy\") pod \"da02a9d7-7b75-401c-9134-8e45d7303175\" (UID: \"da02a9d7-7b75-401c-9134-8e45d7303175\") " Oct 30 19:19:58.738064 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:58.737974 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da02a9d7-7b75-401c-9134-8e45d7303175-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da02a9d7-7b75-401c-9134-8e45d7303175-workload-socket\") pod \"da02a9d7-7b75-401c-9134-8e45d7303175\" (UID: \"da02a9d7-7b75-401c-9134-8e45d7303175\") " Oct 30 19:19:58.738064 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:58.737971 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da02a9d7-7b75-401c-9134-8e45d7303175-istiod-ca-cert" pod "da02a9d7-7b75-401c-9134-8e45d7303175" (UID: "da02a9d7-7b75-401c-9134-8e45d7303175"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:19:58.738064 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:58.737985 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da02a9d7-7b75-401c-9134-8e45d7303175-workload-certs" pod "da02a9d7-7b75-401c-9134-8e45d7303175" (UID: "da02a9d7-7b75-401c-9134-8e45d7303175"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:19:58.738064 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:58.738011 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da02a9d7-7b75-401c-9134-8e45d7303175-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da02a9d7-7b75-401c-9134-8e45d7303175-istio-data\") pod \"da02a9d7-7b75-401c-9134-8e45d7303175\" (UID: \"da02a9d7-7b75-401c-9134-8e45d7303175\") " Oct 30 19:19:58.738357 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:58.738186 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da02a9d7-7b75-401c-9134-8e45d7303175-credential-socket" pod "da02a9d7-7b75-401c-9134-8e45d7303175" (UID: "da02a9d7-7b75-401c-9134-8e45d7303175"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:19:58.738357 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:58.738224 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da02a9d7-7b75-401c-9134-8e45d7303175-workload-socket" pod "da02a9d7-7b75-401c-9134-8e45d7303175" (UID: "da02a9d7-7b75-401c-9134-8e45d7303175"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:19:58.738611 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:58.738585 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da02a9d7-7b75-401c-9134-8e45d7303175-istio-data" pod "da02a9d7-7b75-401c-9134-8e45d7303175" (UID: "da02a9d7-7b75-401c-9134-8e45d7303175"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:19:58.738784 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:58.738760 6960 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da02a9d7-7b75-401c-9134-8e45d7303175-istiod-ca-cert\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:19:58.738910 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:58.738792 6960 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da02a9d7-7b75-401c-9134-8e45d7303175-workload-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:19:58.738910 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:58.738808 6960 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da02a9d7-7b75-401c-9134-8e45d7303175-credential-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:19:58.739044 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:58.738911 6960 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da02a9d7-7b75-401c-9134-8e45d7303175-workload-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:19:58.739044 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:58.739002 6960 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da02a9d7-7b75-401c-9134-8e45d7303175-istio-data\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:19:58.739866 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:58.739844 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da02a9d7-7b75-401c-9134-8e45d7303175-istio-token" pod "da02a9d7-7b75-401c-9134-8e45d7303175" (UID: "da02a9d7-7b75-401c-9134-8e45d7303175"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:19:58.740092 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:58.740076 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/da02a9d7-7b75-401c-9134-8e45d7303175-istio-podinfo" pod "da02a9d7-7b75-401c-9134-8e45d7303175" (UID: "da02a9d7-7b75-401c-9134-8e45d7303175"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:19:58.740422 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:58.740404 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da02a9d7-7b75-401c-9134-8e45d7303175-istio-envoy" pod "da02a9d7-7b75-401c-9134-8e45d7303175" (UID: "da02a9d7-7b75-401c-9134-8e45d7303175"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:19:58.740586 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:58.740566 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da02a9d7-7b75-401c-9134-8e45d7303175-kube-api-access-85wvp" pod "da02a9d7-7b75-401c-9134-8e45d7303175" (UID: "da02a9d7-7b75-401c-9134-8e45d7303175"). InnerVolumeSpecName "kube-api-access-85wvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:19:58.840735 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:58.840700 6960 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da02a9d7-7b75-401c-9134-8e45d7303175-istio-podinfo\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:19:58.840735 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:58.840726 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85wvp\" (UniqueName: \"kubernetes.io/projected/da02a9d7-7b75-401c-9134-8e45d7303175-kube-api-access-85wvp\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:19:58.840735 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:58.840738 6960 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da02a9d7-7b75-401c-9134-8e45d7303175-istio-envoy\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:19:58.841029 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:58.840747 6960 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da02a9d7-7b75-401c-9134-8e45d7303175-istio-token\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:19:59.247070 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:59.247031 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:19:59.312330 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:59.312301 6960 generic.go:356] "Generic (PLEG): container finished" podID="da02a9d7-7b75-401c-9134-8e45d7303175" containerID="30633a687fadded7a1baf9dd9d6a60ebd1a07128924ad553e4a1430a0ccfc5c1" exitCode=0 Oct 30 19:19:59.312509 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:59.312353 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-wfw84" event={"ID":"da02a9d7-7b75-401c-9134-8e45d7303175","Type":"ContainerDied","Data":"30633a687fadded7a1baf9dd9d6a60ebd1a07128924ad553e4a1430a0ccfc5c1"} Oct 30 19:19:59.312509 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:59.312360 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-wfw84" Oct 30 19:19:59.312509 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:59.312394 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-wfw84" event={"ID":"da02a9d7-7b75-401c-9134-8e45d7303175","Type":"ContainerDied","Data":"97cd57819c4c15659434b0a7d7ba5b48343daca34cce1c74ada39251918e515e"} Oct 30 19:19:59.312509 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:59.312415 6960 scope.go:122] "RemoveContainer" containerID="30633a687fadded7a1baf9dd9d6a60ebd1a07128924ad553e4a1430a0ccfc5c1" Oct 30 19:19:59.324163 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:59.324123 6960 scope.go:122] "RemoveContainer" containerID="30633a687fadded7a1baf9dd9d6a60ebd1a07128924ad553e4a1430a0ccfc5c1" Oct 30 19:19:59.324488 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:19:59.324457 6960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30633a687fadded7a1baf9dd9d6a60ebd1a07128924ad553e4a1430a0ccfc5c1\": container with ID starting with 30633a687fadded7a1baf9dd9d6a60ebd1a07128924ad553e4a1430a0ccfc5c1 not found: ID does not exist" containerID="30633a687fadded7a1baf9dd9d6a60ebd1a07128924ad553e4a1430a0ccfc5c1" Oct 30 19:19:59.324625 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:59.324496 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30633a687fadded7a1baf9dd9d6a60ebd1a07128924ad553e4a1430a0ccfc5c1"} err="failed to get container status \"30633a687fadded7a1baf9dd9d6a60ebd1a07128924ad553e4a1430a0ccfc5c1\": rpc error: code = NotFound desc = could not find container \"30633a687fadded7a1baf9dd9d6a60ebd1a07128924ad553e4a1430a0ccfc5c1\": container with ID starting with 30633a687fadded7a1baf9dd9d6a60ebd1a07128924ad553e4a1430a0ccfc5c1 not found: ID does not exist" Oct 30 19:19:59.340982 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:59.340945 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-wfw84"] Oct 30 19:19:59.345604 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:19:59.345575 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-wfw84"] Oct 30 19:20:00.081733 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:00.081697 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:00.594515 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:00.594477 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da02a9d7-7b75-401c-9134-8e45d7303175" path="/var/lib/kubelet/pods/da02a9d7-7b75-401c-9134-8e45d7303175/volumes" Oct 30 19:20:00.914171 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:00.914048 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:01.676342 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:01.676306 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:01.788685 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:01.788643 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:02.107968 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:02.107930 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:03.152496 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:03.152453 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:04.301721 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:04.301690 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:05.103884 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:05.103853 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:05.908195 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:05.908122 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:06.024385 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:06.024354 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:06.835249 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:06.835205 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:08.053698 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:08.053658 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:08.635996 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:08.635963 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:08.754523 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:08.754492 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:08.979225 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:08.979119 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:09.933330 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:09.933294 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:10.720295 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:10.720264 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:11.713670 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:11.713639 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:12.601014 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:12.600980 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:12.971789 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:12.971674 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:13.084285 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:13.084251 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:13.520132 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:13.520093 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:14.331138 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:14.331104 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:15.215351 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:15.215297 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:16.079468 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:16.079436 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:16.676257 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:16.676224 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-4e94aa-57fcc56d98-cpjh8"] Oct 30 19:20:16.676519 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:16.676492 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4e94aa-57fcc56d98-cpjh8" podUID="68ec602e-60f3-4540-bfcf-9b29cdcfda2f" containerName="maas-api" containerID="cri-o://0ff79a08593d1f82155641332c3043c38f9e26a23ac94031466d0441a5de5282" gracePeriod=30 Oct 30 19:20:16.741905 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:16.741864 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:17.161243 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:17.151743 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:17.275000 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:17.274965 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4e94aa-57fcc56d98-cpjh8" Oct 30 19:20:17.356822 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:17.356780 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:17.439873 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:17.439768 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68ec602e-60f3-4540-bfcf-9b29cdcfda2f-kube-api-access-wtx2r\" (UniqueName: \"kubernetes.io/projected/68ec602e-60f3-4540-bfcf-9b29cdcfda2f-kube-api-access-wtx2r\") pod \"68ec602e-60f3-4540-bfcf-9b29cdcfda2f\" (UID: \"68ec602e-60f3-4540-bfcf-9b29cdcfda2f\") " Oct 30 19:20:17.440067 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:17.440044 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68ec602e-60f3-4540-bfcf-9b29cdcfda2f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68ec602e-60f3-4540-bfcf-9b29cdcfda2f-maas-api-tls\") pod \"68ec602e-60f3-4540-bfcf-9b29cdcfda2f\" (UID: \"68ec602e-60f3-4540-bfcf-9b29cdcfda2f\") " Oct 30 19:20:17.440616 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:17.440399 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68ec602e-60f3-4540-bfcf-9b29cdcfda2f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/68ec602e-60f3-4540-bfcf-9b29cdcfda2f-metrics-certs\") pod \"68ec602e-60f3-4540-bfcf-9b29cdcfda2f\" (UID: \"68ec602e-60f3-4540-bfcf-9b29cdcfda2f\") " Oct 30 19:20:17.440616 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:17.440459 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68ec602e-60f3-4540-bfcf-9b29cdcfda2f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68ec602e-60f3-4540-bfcf-9b29cdcfda2f-openshift-service-ca\") pod \"68ec602e-60f3-4540-bfcf-9b29cdcfda2f\" (UID: \"68ec602e-60f3-4540-bfcf-9b29cdcfda2f\") " Oct 30 19:20:17.441340 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:17.441308 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68ec602e-60f3-4540-bfcf-9b29cdcfda2f-openshift-service-ca" pod "68ec602e-60f3-4540-bfcf-9b29cdcfda2f" (UID: "68ec602e-60f3-4540-bfcf-9b29cdcfda2f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:20:17.448218 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:17.447427 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68ec602e-60f3-4540-bfcf-9b29cdcfda2f-maas-api-tls" pod "68ec602e-60f3-4540-bfcf-9b29cdcfda2f" (UID: "68ec602e-60f3-4540-bfcf-9b29cdcfda2f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:20:17.452559 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:17.452434 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68ec602e-60f3-4540-bfcf-9b29cdcfda2f-metrics-certs" pod "68ec602e-60f3-4540-bfcf-9b29cdcfda2f" (UID: "68ec602e-60f3-4540-bfcf-9b29cdcfda2f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:20:17.452883 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:17.452851 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68ec602e-60f3-4540-bfcf-9b29cdcfda2f-kube-api-access-wtx2r" pod "68ec602e-60f3-4540-bfcf-9b29cdcfda2f" (UID: "68ec602e-60f3-4540-bfcf-9b29cdcfda2f"). InnerVolumeSpecName "kube-api-access-wtx2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:20:17.474186 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:17.474115 6960 generic.go:356] "Generic (PLEG): container finished" podID="68ec602e-60f3-4540-bfcf-9b29cdcfda2f" containerID="0ff79a08593d1f82155641332c3043c38f9e26a23ac94031466d0441a5de5282" exitCode=0 Oct 30 19:20:17.474359 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:17.474257 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4e94aa-57fcc56d98-cpjh8" event={"ID":"68ec602e-60f3-4540-bfcf-9b29cdcfda2f","Type":"ContainerDied","Data":"0ff79a08593d1f82155641332c3043c38f9e26a23ac94031466d0441a5de5282"} Oct 30 19:20:17.474359 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:17.474343 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4e94aa-57fcc56d98-cpjh8" event={"ID":"68ec602e-60f3-4540-bfcf-9b29cdcfda2f","Type":"ContainerDied","Data":"32fefe5550c9d2a47f3ae966c487627dcc9602a8746086ed4d48e0f1713127f9"} Oct 30 19:20:17.476171 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:17.474373 6960 scope.go:122] "RemoveContainer" containerID="0ff79a08593d1f82155641332c3043c38f9e26a23ac94031466d0441a5de5282" Oct 30 19:20:17.476171 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:17.474537 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-4e94aa-57fcc56d98-cpjh8" Oct 30 19:20:17.501674 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:17.501641 6960 scope.go:122] "RemoveContainer" containerID="0ff79a08593d1f82155641332c3043c38f9e26a23ac94031466d0441a5de5282" Oct 30 19:20:17.502248 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:17.502217 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:17.502725 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:20:17.502697 6960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ff79a08593d1f82155641332c3043c38f9e26a23ac94031466d0441a5de5282\": container with ID starting with 0ff79a08593d1f82155641332c3043c38f9e26a23ac94031466d0441a5de5282 not found: ID does not exist" containerID="0ff79a08593d1f82155641332c3043c38f9e26a23ac94031466d0441a5de5282" Oct 30 19:20:17.502814 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:17.502736 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ff79a08593d1f82155641332c3043c38f9e26a23ac94031466d0441a5de5282"} err="failed to get container status \"0ff79a08593d1f82155641332c3043c38f9e26a23ac94031466d0441a5de5282\": rpc error: code = NotFound desc = could not find container \"0ff79a08593d1f82155641332c3043c38f9e26a23ac94031466d0441a5de5282\": container with ID starting with 0ff79a08593d1f82155641332c3043c38f9e26a23ac94031466d0441a5de5282 not found: ID does not exist" Oct 30 19:20:17.512514 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:17.512490 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-4e94aa-57fcc56d98-cpjh8"] Oct 30 19:20:17.520081 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:17.520057 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-4e94aa-57fcc56d98-cpjh8"] Oct 30 19:20:17.542640 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:17.542602 6960 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68ec602e-60f3-4540-bfcf-9b29cdcfda2f-metrics-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:17.542812 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:17.542635 6960 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68ec602e-60f3-4540-bfcf-9b29cdcfda2f-openshift-service-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:17.542812 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:17.542678 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtx2r\" (UniqueName: \"kubernetes.io/projected/68ec602e-60f3-4540-bfcf-9b29cdcfda2f-kube-api-access-wtx2r\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:17.542812 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:17.542693 6960 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68ec602e-60f3-4540-bfcf-9b29cdcfda2f-maas-api-tls\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:17.731270 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:17.731175 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:18.433637 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:18.433590 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:18.594159 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:18.594112 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68ec602e-60f3-4540-bfcf-9b29cdcfda2f" path="/var/lib/kubelet/pods/68ec602e-60f3-4540-bfcf-9b29cdcfda2f/volumes" Oct 30 19:20:19.386819 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:19.386777 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:20.225402 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:20.225371 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:21.330660 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:21.330625 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:22.137355 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:22.137324 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:22.991321 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:22.991276 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:23.805243 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:23.805196 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-de4bda-b48fd99d7-rw9vs"] Oct 30 19:20:23.805598 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:23.805560 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-de4bda-b48fd99d7-rw9vs" podUID="aef219d1-2021-4cea-b80b-ed995446c87c" containerName="maas-api" containerID="cri-o://16338eae5e234525201a1f6d81fd58ccf458643ca6bee41dde9c5f375b6d74ce" gracePeriod=30 Oct 30 19:20:23.862228 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:23.862196 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:23.942081 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:20:23.942040 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 19:20:23.971053 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:23.971021 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c3a5b9-787455d59f-kd9ng"] Oct 30 19:20:23.971329 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:23.971275 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c3a5b9-787455d59f-kd9ng" podUID="a59257f8-0671-461d-ac61-4956961be8f8" containerName="maas-api" containerID="cri-o://0ba38578db9019b742e5403c4b2f0a71ef0545854a6b5b2f1ca2d0cc3c4034e1" gracePeriod=30 Oct 30 19:20:24.031092 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:24.031040 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:24.152675 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:24.152430 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:24.215725 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:24.215702 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-de4bda-b48fd99d7-rw9vs" Oct 30 19:20:24.293887 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:24.293842 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aef219d1-2021-4cea-b80b-ed995446c87c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aef219d1-2021-4cea-b80b-ed995446c87c-openshift-service-ca\") pod \"aef219d1-2021-4cea-b80b-ed995446c87c\" (UID: \"aef219d1-2021-4cea-b80b-ed995446c87c\") " Oct 30 19:20:24.317658 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:24.317622 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aef219d1-2021-4cea-b80b-ed995446c87c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/aef219d1-2021-4cea-b80b-ed995446c87c-metrics-certs\") pod \"aef219d1-2021-4cea-b80b-ed995446c87c\" (UID: \"aef219d1-2021-4cea-b80b-ed995446c87c\") " Oct 30 19:20:24.317960 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:24.317941 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aef219d1-2021-4cea-b80b-ed995446c87c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aef219d1-2021-4cea-b80b-ed995446c87c-maas-api-tls\") pod \"aef219d1-2021-4cea-b80b-ed995446c87c\" (UID: \"aef219d1-2021-4cea-b80b-ed995446c87c\") " Oct 30 19:20:24.318114 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:24.318095 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aef219d1-2021-4cea-b80b-ed995446c87c-kube-api-access-kwtzd\" (UniqueName: \"kubernetes.io/projected/aef219d1-2021-4cea-b80b-ed995446c87c-kube-api-access-kwtzd\") pod \"aef219d1-2021-4cea-b80b-ed995446c87c\" (UID: \"aef219d1-2021-4cea-b80b-ed995446c87c\") " Oct 30 19:20:24.346040 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:24.294792 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aef219d1-2021-4cea-b80b-ed995446c87c-openshift-service-ca" pod "aef219d1-2021-4cea-b80b-ed995446c87c" (UID: "aef219d1-2021-4cea-b80b-ed995446c87c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:20:24.376615 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:24.370286 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aef219d1-2021-4cea-b80b-ed995446c87c-maas-api-tls" pod "aef219d1-2021-4cea-b80b-ed995446c87c" (UID: "aef219d1-2021-4cea-b80b-ed995446c87c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:20:24.387195 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:24.382399 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:24.433176 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:24.432559 6960 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aef219d1-2021-4cea-b80b-ed995446c87c-maas-api-tls\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:24.433176 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:24.432601 6960 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aef219d1-2021-4cea-b80b-ed995446c87c-openshift-service-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:24.470218 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:24.469941 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aef219d1-2021-4cea-b80b-ed995446c87c-kube-api-access-kwtzd" pod "aef219d1-2021-4cea-b80b-ed995446c87c" (UID: "aef219d1-2021-4cea-b80b-ed995446c87c"). InnerVolumeSpecName "kube-api-access-kwtzd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:20:24.504818 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:24.504773 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aef219d1-2021-4cea-b80b-ed995446c87c-metrics-certs" pod "aef219d1-2021-4cea-b80b-ed995446c87c" (UID: "aef219d1-2021-4cea-b80b-ed995446c87c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:20:24.551182 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:24.536545 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwtzd\" (UniqueName: \"kubernetes.io/projected/aef219d1-2021-4cea-b80b-ed995446c87c-kube-api-access-kwtzd\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:24.551182 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:24.536573 6960 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aef219d1-2021-4cea-b80b-ed995446c87c-metrics-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:24.561228 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:24.556658 6960 generic.go:356] "Generic (PLEG): container finished" podID="aef219d1-2021-4cea-b80b-ed995446c87c" containerID="16338eae5e234525201a1f6d81fd58ccf458643ca6bee41dde9c5f375b6d74ce" exitCode=0 Oct 30 19:20:24.561228 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:24.556777 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-de4bda-b48fd99d7-rw9vs" event={"ID":"aef219d1-2021-4cea-b80b-ed995446c87c","Type":"ContainerDied","Data":"16338eae5e234525201a1f6d81fd58ccf458643ca6bee41dde9c5f375b6d74ce"} Oct 30 19:20:24.561228 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:24.556825 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-de4bda-b48fd99d7-rw9vs" event={"ID":"aef219d1-2021-4cea-b80b-ed995446c87c","Type":"ContainerDied","Data":"44fe3f4e80c797cf2cde371799bb96ddab1d23b28303569360894697a0c19913"} Oct 30 19:20:24.561228 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:24.556857 6960 scope.go:122] "RemoveContainer" containerID="16338eae5e234525201a1f6d81fd58ccf458643ca6bee41dde9c5f375b6d74ce" Oct 30 19:20:24.561228 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:24.556996 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-de4bda-b48fd99d7-rw9vs" Oct 30 19:20:24.569211 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:24.565399 6960 generic.go:356] "Generic (PLEG): container finished" podID="a59257f8-0671-461d-ac61-4956961be8f8" containerID="0ba38578db9019b742e5403c4b2f0a71ef0545854a6b5b2f1ca2d0cc3c4034e1" exitCode=0 Oct 30 19:20:24.569211 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:24.565481 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c3a5b9-787455d59f-kd9ng" event={"ID":"a59257f8-0671-461d-ac61-4956961be8f8","Type":"ContainerDied","Data":"0ba38578db9019b742e5403c4b2f0a71ef0545854a6b5b2f1ca2d0cc3c4034e1"} Oct 30 19:20:24.588310 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:24.587697 6960 scope.go:122] "RemoveContainer" containerID="16338eae5e234525201a1f6d81fd58ccf458643ca6bee41dde9c5f375b6d74ce" Oct 30 19:20:24.588310 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:20:24.588027 6960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16338eae5e234525201a1f6d81fd58ccf458643ca6bee41dde9c5f375b6d74ce\": container with ID starting with 16338eae5e234525201a1f6d81fd58ccf458643ca6bee41dde9c5f375b6d74ce not found: ID does not exist" containerID="16338eae5e234525201a1f6d81fd58ccf458643ca6bee41dde9c5f375b6d74ce" Oct 30 19:20:24.588310 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:24.588057 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16338eae5e234525201a1f6d81fd58ccf458643ca6bee41dde9c5f375b6d74ce"} err="failed to get container status \"16338eae5e234525201a1f6d81fd58ccf458643ca6bee41dde9c5f375b6d74ce\": rpc error: code = NotFound desc = could not find container \"16338eae5e234525201a1f6d81fd58ccf458643ca6bee41dde9c5f375b6d74ce\": container with ID starting with 16338eae5e234525201a1f6d81fd58ccf458643ca6bee41dde9c5f375b6d74ce not found: ID does not exist" Oct 30 19:20:24.600579 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:24.600556 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-de4bda-b48fd99d7-rw9vs"] Oct 30 19:20:24.601389 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:24.601371 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-de4bda-b48fd99d7-rw9vs"] Oct 30 19:20:24.602789 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:24.602769 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c3a5b9-787455d59f-kd9ng" Oct 30 19:20:24.624893 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:24.624863 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:24.737861 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:24.737776 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:24.739886 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:24.739861 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a59257f8-0671-461d-ac61-4956961be8f8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a59257f8-0671-461d-ac61-4956961be8f8-metrics-certs\") pod \"a59257f8-0671-461d-ac61-4956961be8f8\" (UID: \"a59257f8-0671-461d-ac61-4956961be8f8\") " Oct 30 19:20:24.740052 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:24.740037 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a59257f8-0671-461d-ac61-4956961be8f8-kube-api-access-swbbj\" (UniqueName: \"kubernetes.io/projected/a59257f8-0671-461d-ac61-4956961be8f8-kube-api-access-swbbj\") pod \"a59257f8-0671-461d-ac61-4956961be8f8\" (UID: \"a59257f8-0671-461d-ac61-4956961be8f8\") " Oct 30 19:20:24.740136 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:24.740119 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a59257f8-0671-461d-ac61-4956961be8f8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a59257f8-0671-461d-ac61-4956961be8f8-maas-api-tls\") pod \"a59257f8-0671-461d-ac61-4956961be8f8\" (UID: \"a59257f8-0671-461d-ac61-4956961be8f8\") " Oct 30 19:20:24.740224 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:24.740182 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a59257f8-0671-461d-ac61-4956961be8f8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a59257f8-0671-461d-ac61-4956961be8f8-openshift-service-ca\") pod \"a59257f8-0671-461d-ac61-4956961be8f8\" (UID: \"a59257f8-0671-461d-ac61-4956961be8f8\") " Oct 30 19:20:24.740516 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:24.740489 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a59257f8-0671-461d-ac61-4956961be8f8-openshift-service-ca" pod "a59257f8-0671-461d-ac61-4956961be8f8" (UID: "a59257f8-0671-461d-ac61-4956961be8f8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:20:24.740874 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:24.740830 6960 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a59257f8-0671-461d-ac61-4956961be8f8-openshift-service-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:24.742188 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:24.742166 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a59257f8-0671-461d-ac61-4956961be8f8-metrics-certs" pod "a59257f8-0671-461d-ac61-4956961be8f8" (UID: "a59257f8-0671-461d-ac61-4956961be8f8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:20:24.742470 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:24.742449 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a59257f8-0671-461d-ac61-4956961be8f8-maas-api-tls" pod "a59257f8-0671-461d-ac61-4956961be8f8" (UID: "a59257f8-0671-461d-ac61-4956961be8f8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:20:24.742666 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:24.742649 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a59257f8-0671-461d-ac61-4956961be8f8-kube-api-access-swbbj" pod "a59257f8-0671-461d-ac61-4956961be8f8" (UID: "a59257f8-0671-461d-ac61-4956961be8f8"). InnerVolumeSpecName "kube-api-access-swbbj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:20:24.846179 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:24.843772 6960 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a59257f8-0671-461d-ac61-4956961be8f8-metrics-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:24.846179 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:24.843814 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swbbj\" (UniqueName: \"kubernetes.io/projected/a59257f8-0671-461d-ac61-4956961be8f8-kube-api-access-swbbj\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:24.846179 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:24.843831 6960 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a59257f8-0671-461d-ac61-4956961be8f8-maas-api-tls\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:24.859815 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:24.859087 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:25.230265 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:25.230227 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:25.343950 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:25.343791 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:25.576776 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:25.576749 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c3a5b9-787455d59f-kd9ng" Oct 30 19:20:25.576993 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:25.576749 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c3a5b9-787455d59f-kd9ng" event={"ID":"a59257f8-0671-461d-ac61-4956961be8f8","Type":"ContainerDied","Data":"4da81e17bd88a25e0df4805357c84430c9c124a25f0c89114bc4876713c768af"} Oct 30 19:20:25.577073 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:25.577022 6960 scope.go:122] "RemoveContainer" containerID="0ba38578db9019b742e5403c4b2f0a71ef0545854a6b5b2f1ca2d0cc3c4034e1" Oct 30 19:20:25.623402 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:25.623368 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c3a5b9-787455d59f-kd9ng"] Oct 30 19:20:25.634788 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:25.634755 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-c3a5b9-787455d59f-kd9ng"] Oct 30 19:20:25.695491 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:25.695451 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:25.811170 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:25.805004 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:26.593878 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:26.593847 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a59257f8-0671-461d-ac61-4956961be8f8" path="/var/lib/kubelet/pods/a59257f8-0671-461d-ac61-4956961be8f8/volumes" Oct 30 19:20:26.594552 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:26.594530 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aef219d1-2021-4cea-b80b-ed995446c87c" path="/var/lib/kubelet/pods/aef219d1-2021-4cea-b80b-ed995446c87c/volumes" Oct 30 19:20:26.808350 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:26.808311 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:27.766459 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:27.766424 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:28.538208 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:28.538177 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:29.401739 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:29.401708 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:30.377613 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:30.377580 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:31.315758 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:31.315723 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:32.100913 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:32.100877 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:32.814917 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:32.814868 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:32.966462 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:32.966425 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:33.148046 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:33.147955 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-fln48"] Oct 30 19:20:33.148293 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:33.148228 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-fln48" podUID="904f96ca-921a-4d2e-8245-46500d6b7d0a" containerName="istio-proxy" containerID="cri-o://9a77e88de0d86c0fb40f91c3e22c86a41c5d36c6642a1b4deb273aa333fc1250" gracePeriod=30 Oct 30 19:20:33.187825 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:33.187791 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:33.389801 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:33.389765 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:33.614207 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:33.614173 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:33.806577 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:33.806542 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-qhvj4"] Oct 30 19:20:33.806789 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:33.806754 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-qhvj4" podUID="b06be103-9039-4229-bb04-d464932e988f" containerName="istio-proxy" containerID="cri-o://fcbb28957b6101db73040aae408c3745b8d1854c2fcb157b2522b30ed6ddf45b" gracePeriod=30 Oct 30 19:20:33.819293 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:33.819265 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-x95gb"] Oct 30 19:20:33.819659 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:33.819441 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-x95gb" podUID="94b983c1-08fc-4f62-85de-e24696834c7c" containerName="istio-proxy" containerID="cri-o://2f742c3a4ba0c9b04c13be9851d0c9eaa29e63a527939ae7663cfd119a048c66" gracePeriod=30 Oct 30 19:20:33.862075 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:33.862044 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:34.263354 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:34.263323 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:35.305521 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:35.305481 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:36.219054 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:36.219021 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:36.428944 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:36.428909 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:36.547761 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:36.547665 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:37.166920 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:37.166871 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:37.554603 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:37.554575 6960 patch_prober.go:29] interesting pod/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-x95gb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:20:37.554986 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:37.554637 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-x95gb" podUID="94b983c1-08fc-4f62-85de-e24696834c7c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:20:37.668636 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:37.668606 6960 patch_prober.go:29] interesting pod/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-qhvj4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:20:37.668826 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:37.668670 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-qhvj4" podUID="b06be103-9039-4229-bb04-d464932e988f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:20:37.971045 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:37.966885 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:38.452947 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:38.452921 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-fln48" Oct 30 19:20:38.512411 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:38.512381 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/904f96ca-921a-4d2e-8245-46500d6b7d0a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/904f96ca-921a-4d2e-8245-46500d6b7d0a-credential-socket\") pod \"904f96ca-921a-4d2e-8245-46500d6b7d0a\" (UID: \"904f96ca-921a-4d2e-8245-46500d6b7d0a\") " Oct 30 19:20:38.512596 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:38.512435 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/904f96ca-921a-4d2e-8245-46500d6b7d0a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/904f96ca-921a-4d2e-8245-46500d6b7d0a-istio-envoy\") pod \"904f96ca-921a-4d2e-8245-46500d6b7d0a\" (UID: \"904f96ca-921a-4d2e-8245-46500d6b7d0a\") " Oct 30 19:20:38.512596 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:38.512512 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/904f96ca-921a-4d2e-8245-46500d6b7d0a-kube-api-access-txjt7\" (UniqueName: \"kubernetes.io/projected/904f96ca-921a-4d2e-8245-46500d6b7d0a-kube-api-access-txjt7\") pod \"904f96ca-921a-4d2e-8245-46500d6b7d0a\" (UID: \"904f96ca-921a-4d2e-8245-46500d6b7d0a\") " Oct 30 19:20:38.512596 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:38.512577 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/904f96ca-921a-4d2e-8245-46500d6b7d0a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/904f96ca-921a-4d2e-8245-46500d6b7d0a-istiod-ca-cert\") pod \"904f96ca-921a-4d2e-8245-46500d6b7d0a\" (UID: \"904f96ca-921a-4d2e-8245-46500d6b7d0a\") " Oct 30 19:20:38.512718 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:38.512651 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/904f96ca-921a-4d2e-8245-46500d6b7d0a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/904f96ca-921a-4d2e-8245-46500d6b7d0a-istio-data\") pod \"904f96ca-921a-4d2e-8245-46500d6b7d0a\" (UID: \"904f96ca-921a-4d2e-8245-46500d6b7d0a\") " Oct 30 19:20:38.512718 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:38.512684 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/904f96ca-921a-4d2e-8245-46500d6b7d0a-credential-socket" pod "904f96ca-921a-4d2e-8245-46500d6b7d0a" (UID: "904f96ca-921a-4d2e-8245-46500d6b7d0a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:20:38.512841 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:38.512737 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/904f96ca-921a-4d2e-8245-46500d6b7d0a-istio-token\" (UniqueName: \"kubernetes.io/projected/904f96ca-921a-4d2e-8245-46500d6b7d0a-istio-token\") pod \"904f96ca-921a-4d2e-8245-46500d6b7d0a\" (UID: \"904f96ca-921a-4d2e-8245-46500d6b7d0a\") " Oct 30 19:20:38.512841 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:38.512788 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/904f96ca-921a-4d2e-8245-46500d6b7d0a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/904f96ca-921a-4d2e-8245-46500d6b7d0a-workload-certs\") pod \"904f96ca-921a-4d2e-8245-46500d6b7d0a\" (UID: \"904f96ca-921a-4d2e-8245-46500d6b7d0a\") " Oct 30 19:20:38.512916 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:38.512867 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/904f96ca-921a-4d2e-8245-46500d6b7d0a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/904f96ca-921a-4d2e-8245-46500d6b7d0a-istio-podinfo\") pod \"904f96ca-921a-4d2e-8245-46500d6b7d0a\" (UID: \"904f96ca-921a-4d2e-8245-46500d6b7d0a\") " Oct 30 19:20:38.512916 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:38.512902 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/904f96ca-921a-4d2e-8245-46500d6b7d0a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/904f96ca-921a-4d2e-8245-46500d6b7d0a-workload-socket\") pod \"904f96ca-921a-4d2e-8245-46500d6b7d0a\" (UID: \"904f96ca-921a-4d2e-8245-46500d6b7d0a\") " Oct 30 19:20:38.513073 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:38.512867 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/904f96ca-921a-4d2e-8245-46500d6b7d0a-istio-data" pod "904f96ca-921a-4d2e-8245-46500d6b7d0a" (UID: "904f96ca-921a-4d2e-8245-46500d6b7d0a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:20:38.513344 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:38.513318 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/904f96ca-921a-4d2e-8245-46500d6b7d0a-workload-certs" pod "904f96ca-921a-4d2e-8245-46500d6b7d0a" (UID: "904f96ca-921a-4d2e-8245-46500d6b7d0a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:20:38.513532 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:38.513515 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/904f96ca-921a-4d2e-8245-46500d6b7d0a-workload-socket" pod "904f96ca-921a-4d2e-8245-46500d6b7d0a" (UID: "904f96ca-921a-4d2e-8245-46500d6b7d0a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:20:38.513615 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:38.513451 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/904f96ca-921a-4d2e-8245-46500d6b7d0a-istiod-ca-cert" pod "904f96ca-921a-4d2e-8245-46500d6b7d0a" (UID: "904f96ca-921a-4d2e-8245-46500d6b7d0a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:20:38.513615 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:38.513562 6960 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/904f96ca-921a-4d2e-8245-46500d6b7d0a-workload-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:38.513615 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:38.513584 6960 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/904f96ca-921a-4d2e-8245-46500d6b7d0a-credential-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:38.513615 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:38.513598 6960 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/904f96ca-921a-4d2e-8245-46500d6b7d0a-istio-data\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:38.515996 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:38.515954 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/904f96ca-921a-4d2e-8245-46500d6b7d0a-istio-token" pod "904f96ca-921a-4d2e-8245-46500d6b7d0a" (UID: "904f96ca-921a-4d2e-8245-46500d6b7d0a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:20:38.516124 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:38.516068 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/904f96ca-921a-4d2e-8245-46500d6b7d0a-istio-envoy" pod "904f96ca-921a-4d2e-8245-46500d6b7d0a" (UID: "904f96ca-921a-4d2e-8245-46500d6b7d0a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:20:38.516124 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:38.516096 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/904f96ca-921a-4d2e-8245-46500d6b7d0a-istio-podinfo" pod "904f96ca-921a-4d2e-8245-46500d6b7d0a" (UID: "904f96ca-921a-4d2e-8245-46500d6b7d0a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:20:38.516401 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:38.516367 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/904f96ca-921a-4d2e-8245-46500d6b7d0a-kube-api-access-txjt7" pod "904f96ca-921a-4d2e-8245-46500d6b7d0a" (UID: "904f96ca-921a-4d2e-8245-46500d6b7d0a"). InnerVolumeSpecName "kube-api-access-txjt7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:20:38.614562 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:38.614535 6960 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/904f96ca-921a-4d2e-8245-46500d6b7d0a-istio-podinfo\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:38.614562 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:38.614559 6960 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/904f96ca-921a-4d2e-8245-46500d6b7d0a-workload-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:38.614562 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:38.614569 6960 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/904f96ca-921a-4d2e-8245-46500d6b7d0a-istio-envoy\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:38.614562 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:38.614577 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txjt7\" (UniqueName: \"kubernetes.io/projected/904f96ca-921a-4d2e-8245-46500d6b7d0a-kube-api-access-txjt7\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:38.615052 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:38.614587 6960 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/904f96ca-921a-4d2e-8245-46500d6b7d0a-istiod-ca-cert\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:38.615052 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:38.614600 6960 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/904f96ca-921a-4d2e-8245-46500d6b7d0a-istio-token\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:38.699677 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:38.699640 6960 generic.go:356] "Generic (PLEG): container finished" podID="904f96ca-921a-4d2e-8245-46500d6b7d0a" containerID="9a77e88de0d86c0fb40f91c3e22c86a41c5d36c6642a1b4deb273aa333fc1250" exitCode=0 Oct 30 19:20:38.699863 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:38.699682 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-fln48" Oct 30 19:20:38.699863 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:38.699721 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-fln48" event={"ID":"904f96ca-921a-4d2e-8245-46500d6b7d0a","Type":"ContainerDied","Data":"9a77e88de0d86c0fb40f91c3e22c86a41c5d36c6642a1b4deb273aa333fc1250"} Oct 30 19:20:38.699863 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:38.699798 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-fln48" event={"ID":"904f96ca-921a-4d2e-8245-46500d6b7d0a","Type":"ContainerDied","Data":"ff244b47abee055e9f802b2bb6d5dbf2b78afbcf27e19e27743d91ff4d863ce4"} Oct 30 19:20:38.699863 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:38.699830 6960 scope.go:122] "RemoveContainer" containerID="9a77e88de0d86c0fb40f91c3e22c86a41c5d36c6642a1b4deb273aa333fc1250" Oct 30 19:20:38.711620 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:38.711600 6960 scope.go:122] "RemoveContainer" containerID="9a77e88de0d86c0fb40f91c3e22c86a41c5d36c6642a1b4deb273aa333fc1250" Oct 30 19:20:38.711905 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:20:38.711880 6960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a77e88de0d86c0fb40f91c3e22c86a41c5d36c6642a1b4deb273aa333fc1250\": container with ID starting with 9a77e88de0d86c0fb40f91c3e22c86a41c5d36c6642a1b4deb273aa333fc1250 not found: ID does not exist" containerID="9a77e88de0d86c0fb40f91c3e22c86a41c5d36c6642a1b4deb273aa333fc1250" Oct 30 19:20:38.711977 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:38.711918 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a77e88de0d86c0fb40f91c3e22c86a41c5d36c6642a1b4deb273aa333fc1250"} err="failed to get container status \"9a77e88de0d86c0fb40f91c3e22c86a41c5d36c6642a1b4deb273aa333fc1250\": rpc error: code = NotFound desc = could not find container \"9a77e88de0d86c0fb40f91c3e22c86a41c5d36c6642a1b4deb273aa333fc1250\": container with ID starting with 9a77e88de0d86c0fb40f91c3e22c86a41c5d36c6642a1b4deb273aa333fc1250 not found: ID does not exist" Oct 30 19:20:38.722422 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:38.722402 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-fln48"] Oct 30 19:20:38.729531 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:38.729492 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:38.729692 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:38.729673 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-c3a5b9-openshift-default-58f8555bc9-fln48"] Oct 30 19:20:39.072773 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:39.072709 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-qhvj4" Oct 30 19:20:39.124857 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:39.124826 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b06be103-9039-4229-bb04-d464932e988f-kube-api-access-b6t94\" (UniqueName: \"kubernetes.io/projected/b06be103-9039-4229-bb04-d464932e988f-kube-api-access-b6t94\") pod \"b06be103-9039-4229-bb04-d464932e988f\" (UID: \"b06be103-9039-4229-bb04-d464932e988f\") " Oct 30 19:20:39.125029 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:39.124999 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b06be103-9039-4229-bb04-d464932e988f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b06be103-9039-4229-bb04-d464932e988f-credential-socket\") pod \"b06be103-9039-4229-bb04-d464932e988f\" (UID: \"b06be103-9039-4229-bb04-d464932e988f\") " Oct 30 19:20:39.125091 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:39.125039 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b06be103-9039-4229-bb04-d464932e988f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b06be103-9039-4229-bb04-d464932e988f-workload-certs\") pod \"b06be103-9039-4229-bb04-d464932e988f\" (UID: \"b06be103-9039-4229-bb04-d464932e988f\") " Oct 30 19:20:39.125191 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:39.125097 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b06be103-9039-4229-bb04-d464932e988f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b06be103-9039-4229-bb04-d464932e988f-istio-data\") pod \"b06be103-9039-4229-bb04-d464932e988f\" (UID: \"b06be103-9039-4229-bb04-d464932e988f\") " Oct 30 19:20:39.125191 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:39.125183 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b06be103-9039-4229-bb04-d464932e988f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b06be103-9039-4229-bb04-d464932e988f-istio-podinfo\") pod \"b06be103-9039-4229-bb04-d464932e988f\" (UID: \"b06be103-9039-4229-bb04-d464932e988f\") " Oct 30 19:20:39.125426 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:39.125221 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b06be103-9039-4229-bb04-d464932e988f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b06be103-9039-4229-bb04-d464932e988f-istiod-ca-cert\") pod \"b06be103-9039-4229-bb04-d464932e988f\" (UID: \"b06be103-9039-4229-bb04-d464932e988f\") " Oct 30 19:20:39.125426 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:39.125271 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b06be103-9039-4229-bb04-d464932e988f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b06be103-9039-4229-bb04-d464932e988f-workload-socket\") pod \"b06be103-9039-4229-bb04-d464932e988f\" (UID: \"b06be103-9039-4229-bb04-d464932e988f\") " Oct 30 19:20:39.125426 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:39.125305 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b06be103-9039-4229-bb04-d464932e988f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b06be103-9039-4229-bb04-d464932e988f-istio-envoy\") pod \"b06be103-9039-4229-bb04-d464932e988f\" (UID: \"b06be103-9039-4229-bb04-d464932e988f\") " Oct 30 19:20:39.125426 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:39.125377 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b06be103-9039-4229-bb04-d464932e988f-istio-token\" (UniqueName: \"kubernetes.io/projected/b06be103-9039-4229-bb04-d464932e988f-istio-token\") pod \"b06be103-9039-4229-bb04-d464932e988f\" (UID: \"b06be103-9039-4229-bb04-d464932e988f\") " Oct 30 19:20:39.125654 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:39.125528 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b06be103-9039-4229-bb04-d464932e988f-istio-data" pod "b06be103-9039-4229-bb04-d464932e988f" (UID: "b06be103-9039-4229-bb04-d464932e988f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:20:39.125863 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:39.125814 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b06be103-9039-4229-bb04-d464932e988f-workload-socket" pod "b06be103-9039-4229-bb04-d464932e988f" (UID: "b06be103-9039-4229-bb04-d464932e988f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:20:39.126235 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:39.126171 6960 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b06be103-9039-4229-bb04-d464932e988f-istio-data\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:39.126235 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:39.126198 6960 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b06be103-9039-4229-bb04-d464932e988f-workload-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:39.126788 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:39.126751 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b06be103-9039-4229-bb04-d464932e988f-istiod-ca-cert" pod "b06be103-9039-4229-bb04-d464932e988f" (UID: "b06be103-9039-4229-bb04-d464932e988f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:20:39.126903 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:39.126856 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b06be103-9039-4229-bb04-d464932e988f-workload-certs" pod "b06be103-9039-4229-bb04-d464932e988f" (UID: "b06be103-9039-4229-bb04-d464932e988f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:20:39.127316 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:39.127290 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b06be103-9039-4229-bb04-d464932e988f-credential-socket" pod "b06be103-9039-4229-bb04-d464932e988f" (UID: "b06be103-9039-4229-bb04-d464932e988f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:20:39.128328 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:39.128301 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b06be103-9039-4229-bb04-d464932e988f-istio-podinfo" pod "b06be103-9039-4229-bb04-d464932e988f" (UID: "b06be103-9039-4229-bb04-d464932e988f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:20:39.128432 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:39.128393 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b06be103-9039-4229-bb04-d464932e988f-kube-api-access-b6t94" pod "b06be103-9039-4229-bb04-d464932e988f" (UID: "b06be103-9039-4229-bb04-d464932e988f"). InnerVolumeSpecName "kube-api-access-b6t94". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:20:39.129257 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:39.129223 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b06be103-9039-4229-bb04-d464932e988f-istio-envoy" pod "b06be103-9039-4229-bb04-d464932e988f" (UID: "b06be103-9039-4229-bb04-d464932e988f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:20:39.131412 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:39.131391 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b06be103-9039-4229-bb04-d464932e988f-istio-token" pod "b06be103-9039-4229-bb04-d464932e988f" (UID: "b06be103-9039-4229-bb04-d464932e988f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:20:39.157314 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:39.157290 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-x95gb" Oct 30 19:20:39.227700 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:39.227239 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94b983c1-08fc-4f62-85de-e24696834c7c-istio-token\" (UniqueName: \"kubernetes.io/projected/94b983c1-08fc-4f62-85de-e24696834c7c-istio-token\") pod \"94b983c1-08fc-4f62-85de-e24696834c7c\" (UID: \"94b983c1-08fc-4f62-85de-e24696834c7c\") " Oct 30 19:20:39.227700 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:39.227332 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/94b983c1-08fc-4f62-85de-e24696834c7c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94b983c1-08fc-4f62-85de-e24696834c7c-istio-podinfo\") pod \"94b983c1-08fc-4f62-85de-e24696834c7c\" (UID: \"94b983c1-08fc-4f62-85de-e24696834c7c\") " Oct 30 19:20:39.227700 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:39.227365 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94b983c1-08fc-4f62-85de-e24696834c7c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94b983c1-08fc-4f62-85de-e24696834c7c-workload-socket\") pod \"94b983c1-08fc-4f62-85de-e24696834c7c\" (UID: \"94b983c1-08fc-4f62-85de-e24696834c7c\") " Oct 30 19:20:39.227700 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:39.227458 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94b983c1-08fc-4f62-85de-e24696834c7c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94b983c1-08fc-4f62-85de-e24696834c7c-istio-envoy\") pod \"94b983c1-08fc-4f62-85de-e24696834c7c\" (UID: \"94b983c1-08fc-4f62-85de-e24696834c7c\") " Oct 30 19:20:39.228403 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:39.228375 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94b983c1-08fc-4f62-85de-e24696834c7c-credential-socket" pod "94b983c1-08fc-4f62-85de-e24696834c7c" (UID: "94b983c1-08fc-4f62-85de-e24696834c7c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:20:39.228403 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:39.227786 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94b983c1-08fc-4f62-85de-e24696834c7c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94b983c1-08fc-4f62-85de-e24696834c7c-credential-socket\") pod \"94b983c1-08fc-4f62-85de-e24696834c7c\" (UID: \"94b983c1-08fc-4f62-85de-e24696834c7c\") " Oct 30 19:20:39.228584 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:39.228551 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94b983c1-08fc-4f62-85de-e24696834c7c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94b983c1-08fc-4f62-85de-e24696834c7c-istiod-ca-cert\") pod \"94b983c1-08fc-4f62-85de-e24696834c7c\" (UID: \"94b983c1-08fc-4f62-85de-e24696834c7c\") " Oct 30 19:20:39.228870 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:39.228844 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94b983c1-08fc-4f62-85de-e24696834c7c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94b983c1-08fc-4f62-85de-e24696834c7c-workload-certs\") pod \"94b983c1-08fc-4f62-85de-e24696834c7c\" (UID: \"94b983c1-08fc-4f62-85de-e24696834c7c\") " Oct 30 19:20:39.230166 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:39.228910 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94b983c1-08fc-4f62-85de-e24696834c7c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94b983c1-08fc-4f62-85de-e24696834c7c-istio-data\") pod \"94b983c1-08fc-4f62-85de-e24696834c7c\" (UID: \"94b983c1-08fc-4f62-85de-e24696834c7c\") " Oct 30 19:20:39.230166 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:39.228968 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94b983c1-08fc-4f62-85de-e24696834c7c-kube-api-access-5cqq4\" (UniqueName: \"kubernetes.io/projected/94b983c1-08fc-4f62-85de-e24696834c7c-kube-api-access-5cqq4\") pod \"94b983c1-08fc-4f62-85de-e24696834c7c\" (UID: \"94b983c1-08fc-4f62-85de-e24696834c7c\") " Oct 30 19:20:39.230166 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:39.229925 6960 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b06be103-9039-4229-bb04-d464932e988f-istio-podinfo\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:39.230166 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:39.229949 6960 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b06be103-9039-4229-bb04-d464932e988f-istiod-ca-cert\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:39.230166 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:39.229964 6960 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b06be103-9039-4229-bb04-d464932e988f-istio-envoy\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:39.230166 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:39.229977 6960 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b06be103-9039-4229-bb04-d464932e988f-istio-token\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:39.230166 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:39.229993 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6t94\" (UniqueName: \"kubernetes.io/projected/b06be103-9039-4229-bb04-d464932e988f-kube-api-access-b6t94\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:39.230166 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:39.230007 6960 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94b983c1-08fc-4f62-85de-e24696834c7c-credential-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:39.230166 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:39.230020 6960 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b06be103-9039-4229-bb04-d464932e988f-credential-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:39.230166 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:39.230034 6960 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b06be103-9039-4229-bb04-d464932e988f-workload-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:39.233489 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:39.232998 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94b983c1-08fc-4f62-85de-e24696834c7c-istio-token" pod "94b983c1-08fc-4f62-85de-e24696834c7c" (UID: "94b983c1-08fc-4f62-85de-e24696834c7c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:20:39.233489 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:39.233245 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94b983c1-08fc-4f62-85de-e24696834c7c-workload-socket" pod "94b983c1-08fc-4f62-85de-e24696834c7c" (UID: "94b983c1-08fc-4f62-85de-e24696834c7c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:20:39.236132 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:39.236106 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94b983c1-08fc-4f62-85de-e24696834c7c-workload-certs" pod "94b983c1-08fc-4f62-85de-e24696834c7c" (UID: "94b983c1-08fc-4f62-85de-e24696834c7c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:20:39.236278 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:39.236249 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94b983c1-08fc-4f62-85de-e24696834c7c-istio-data" pod "94b983c1-08fc-4f62-85de-e24696834c7c" (UID: "94b983c1-08fc-4f62-85de-e24696834c7c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:20:39.236373 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:39.236294 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94b983c1-08fc-4f62-85de-e24696834c7c-istiod-ca-cert" pod "94b983c1-08fc-4f62-85de-e24696834c7c" (UID: "94b983c1-08fc-4f62-85de-e24696834c7c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:20:39.239664 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:39.239584 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/94b983c1-08fc-4f62-85de-e24696834c7c-istio-podinfo" pod "94b983c1-08fc-4f62-85de-e24696834c7c" (UID: "94b983c1-08fc-4f62-85de-e24696834c7c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:20:39.240037 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:39.240000 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94b983c1-08fc-4f62-85de-e24696834c7c-kube-api-access-5cqq4" pod "94b983c1-08fc-4f62-85de-e24696834c7c" (UID: "94b983c1-08fc-4f62-85de-e24696834c7c"). InnerVolumeSpecName "kube-api-access-5cqq4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:20:39.242384 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:39.240480 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94b983c1-08fc-4f62-85de-e24696834c7c-istio-envoy" pod "94b983c1-08fc-4f62-85de-e24696834c7c" (UID: "94b983c1-08fc-4f62-85de-e24696834c7c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:20:39.331011 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:39.330942 6960 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94b983c1-08fc-4f62-85de-e24696834c7c-istio-data\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:39.331011 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:39.330969 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cqq4\" (UniqueName: \"kubernetes.io/projected/94b983c1-08fc-4f62-85de-e24696834c7c-kube-api-access-5cqq4\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:39.331011 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:39.330980 6960 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94b983c1-08fc-4f62-85de-e24696834c7c-istio-token\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:39.331011 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:39.330988 6960 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94b983c1-08fc-4f62-85de-e24696834c7c-istio-podinfo\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:39.331011 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:39.330997 6960 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94b983c1-08fc-4f62-85de-e24696834c7c-workload-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:39.331011 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:39.331006 6960 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94b983c1-08fc-4f62-85de-e24696834c7c-istio-envoy\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:39.331011 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:39.331014 6960 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94b983c1-08fc-4f62-85de-e24696834c7c-istiod-ca-cert\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:39.331377 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:39.331022 6960 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94b983c1-08fc-4f62-85de-e24696834c7c-workload-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:39.692999 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:39.692907 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:39.707927 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:39.707891 6960 generic.go:356] "Generic (PLEG): container finished" podID="b06be103-9039-4229-bb04-d464932e988f" containerID="fcbb28957b6101db73040aae408c3745b8d1854c2fcb157b2522b30ed6ddf45b" exitCode=0 Oct 30 19:20:39.708081 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:39.707936 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-qhvj4" Oct 30 19:20:39.708081 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:39.707983 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-qhvj4" event={"ID":"b06be103-9039-4229-bb04-d464932e988f","Type":"ContainerDied","Data":"fcbb28957b6101db73040aae408c3745b8d1854c2fcb157b2522b30ed6ddf45b"} Oct 30 19:20:39.708081 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:39.708039 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-qhvj4" event={"ID":"b06be103-9039-4229-bb04-d464932e988f","Type":"ContainerDied","Data":"264b0054f2853d993e9d94b8c86a8083a75a97063f6ae4be975b6a5f2db95dbf"} Oct 30 19:20:39.708081 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:39.708066 6960 scope.go:122] "RemoveContainer" containerID="fcbb28957b6101db73040aae408c3745b8d1854c2fcb157b2522b30ed6ddf45b" Oct 30 19:20:39.710087 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:39.709729 6960 generic.go:356] "Generic (PLEG): container finished" podID="94b983c1-08fc-4f62-85de-e24696834c7c" containerID="2f742c3a4ba0c9b04c13be9851d0c9eaa29e63a527939ae7663cfd119a048c66" exitCode=0 Oct 30 19:20:39.710087 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:39.709824 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-x95gb" Oct 30 19:20:39.710087 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:39.709905 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-x95gb" event={"ID":"94b983c1-08fc-4f62-85de-e24696834c7c","Type":"ContainerDied","Data":"2f742c3a4ba0c9b04c13be9851d0c9eaa29e63a527939ae7663cfd119a048c66"} Oct 30 19:20:39.710087 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:39.709947 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-x95gb" event={"ID":"94b983c1-08fc-4f62-85de-e24696834c7c","Type":"ContainerDied","Data":"d03f8f09adcde2ed208f1fc13fe18aeb9e67a404147df5f815432504ccc5c115"} Oct 30 19:20:39.721555 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:39.721540 6960 scope.go:122] "RemoveContainer" containerID="fcbb28957b6101db73040aae408c3745b8d1854c2fcb157b2522b30ed6ddf45b" Oct 30 19:20:39.721787 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:20:39.721770 6960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcbb28957b6101db73040aae408c3745b8d1854c2fcb157b2522b30ed6ddf45b\": container with ID starting with fcbb28957b6101db73040aae408c3745b8d1854c2fcb157b2522b30ed6ddf45b not found: ID does not exist" containerID="fcbb28957b6101db73040aae408c3745b8d1854c2fcb157b2522b30ed6ddf45b" Oct 30 19:20:39.721828 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:39.721795 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcbb28957b6101db73040aae408c3745b8d1854c2fcb157b2522b30ed6ddf45b"} err="failed to get container status \"fcbb28957b6101db73040aae408c3745b8d1854c2fcb157b2522b30ed6ddf45b\": rpc error: code = NotFound desc = could not find container \"fcbb28957b6101db73040aae408c3745b8d1854c2fcb157b2522b30ed6ddf45b\": container with ID starting with fcbb28957b6101db73040aae408c3745b8d1854c2fcb157b2522b30ed6ddf45b not found: ID does not exist" Oct 30 19:20:39.721828 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:39.721810 6960 scope.go:122] "RemoveContainer" containerID="2f742c3a4ba0c9b04c13be9851d0c9eaa29e63a527939ae7663cfd119a048c66" Oct 30 19:20:39.746092 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:39.745011 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-qhvj4"] Oct 30 19:20:39.765127 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:39.765094 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-qhvj4"] Oct 30 19:20:39.769373 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:39.768982 6960 scope.go:122] "RemoveContainer" containerID="2f742c3a4ba0c9b04c13be9851d0c9eaa29e63a527939ae7663cfd119a048c66" Oct 30 19:20:39.769467 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:20:39.769436 6960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f742c3a4ba0c9b04c13be9851d0c9eaa29e63a527939ae7663cfd119a048c66\": container with ID starting with 2f742c3a4ba0c9b04c13be9851d0c9eaa29e63a527939ae7663cfd119a048c66 not found: ID does not exist" containerID="2f742c3a4ba0c9b04c13be9851d0c9eaa29e63a527939ae7663cfd119a048c66" Oct 30 19:20:39.769522 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:39.769474 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f742c3a4ba0c9b04c13be9851d0c9eaa29e63a527939ae7663cfd119a048c66"} err="failed to get container status \"2f742c3a4ba0c9b04c13be9851d0c9eaa29e63a527939ae7663cfd119a048c66\": rpc error: code = NotFound desc = could not find container \"2f742c3a4ba0c9b04c13be9851d0c9eaa29e63a527939ae7663cfd119a048c66\": container with ID starting with 2f742c3a4ba0c9b04c13be9851d0c9eaa29e63a527939ae7663cfd119a048c66 not found: ID does not exist" Oct 30 19:20:39.771774 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:39.771751 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-x95gb"] Oct 30 19:20:39.775968 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:39.775944 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-de4bda-openshift-default-5766cd8c57-x95gb"] Oct 30 19:20:40.461339 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:40.461301 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:40.595320 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:40.595274 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="904f96ca-921a-4d2e-8245-46500d6b7d0a" path="/var/lib/kubelet/pods/904f96ca-921a-4d2e-8245-46500d6b7d0a/volumes" Oct 30 19:20:40.596122 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:40.596100 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94b983c1-08fc-4f62-85de-e24696834c7c" path="/var/lib/kubelet/pods/94b983c1-08fc-4f62-85de-e24696834c7c/volumes" Oct 30 19:20:40.596916 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:40.596894 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b06be103-9039-4229-bb04-d464932e988f" path="/var/lib/kubelet/pods/b06be103-9039-4229-bb04-d464932e988f/volumes" Oct 30 19:20:41.331067 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:41.331030 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:42.132547 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:42.132505 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:43.001726 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:43.001688 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:43.689562 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:43.689525 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:43.971117 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:43.971008 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:44.162411 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:44.162379 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:44.592913 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:44.592881 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:44.744391 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:44.744359 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:45.276775 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:45.276735 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:46.404294 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:46.404238 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:47.445488 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:47.445442 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:48.214372 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:48.214333 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-7c6027-7b8d4f4fb9-mpf4x"] Oct 30 19:20:48.214640 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:48.214613 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7c6027-7b8d4f4fb9-mpf4x" podUID="cd525e12-1c0c-4ab5-91ac-9028a847cab6" containerName="maas-api" containerID="cri-o://8941d022aae24244d7adb90695623dcb4e2427f5ecf04a4363c40ead06aaaf8e" gracePeriod=30 Oct 30 19:20:48.228122 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:48.228090 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7c6027-7b8d4f4fb9-mpf4x" podUID="cd525e12-1c0c-4ab5-91ac-9028a847cab6" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.132:8443/health\": dial tcp 10.134.0.132:8443: connect: connection refused" Oct 30 19:20:48.304685 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:48.304632 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:48.622457 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:48.622415 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:48.644981 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:48.644956 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7c6027-7b8d4f4fb9-mpf4x" Oct 30 19:20:48.738624 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:48.738588 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:48.780572 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:48.780532 6960 generic.go:356] "Generic (PLEG): container finished" podID="cd525e12-1c0c-4ab5-91ac-9028a847cab6" containerID="8941d022aae24244d7adb90695623dcb4e2427f5ecf04a4363c40ead06aaaf8e" exitCode=0 Oct 30 19:20:48.780770 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:48.780581 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7c6027-7b8d4f4fb9-mpf4x" Oct 30 19:20:48.780770 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:48.780649 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7c6027-7b8d4f4fb9-mpf4x" event={"ID":"cd525e12-1c0c-4ab5-91ac-9028a847cab6","Type":"ContainerDied","Data":"8941d022aae24244d7adb90695623dcb4e2427f5ecf04a4363c40ead06aaaf8e"} Oct 30 19:20:48.780770 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:48.780700 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7c6027-7b8d4f4fb9-mpf4x" event={"ID":"cd525e12-1c0c-4ab5-91ac-9028a847cab6","Type":"ContainerDied","Data":"8033161bcd4e953e4c574934874cd27fae4e4a0c0b04be4d0b6b6e866a51c0c5"} Oct 30 19:20:48.780770 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:48.780727 6960 scope.go:122] "RemoveContainer" containerID="8941d022aae24244d7adb90695623dcb4e2427f5ecf04a4363c40ead06aaaf8e" Oct 30 19:20:48.791872 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:48.791853 6960 scope.go:122] "RemoveContainer" containerID="8941d022aae24244d7adb90695623dcb4e2427f5ecf04a4363c40ead06aaaf8e" Oct 30 19:20:48.792180 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:20:48.792136 6960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8941d022aae24244d7adb90695623dcb4e2427f5ecf04a4363c40ead06aaaf8e\": container with ID starting with 8941d022aae24244d7adb90695623dcb4e2427f5ecf04a4363c40ead06aaaf8e not found: ID does not exist" containerID="8941d022aae24244d7adb90695623dcb4e2427f5ecf04a4363c40ead06aaaf8e" Oct 30 19:20:48.792274 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:48.792192 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8941d022aae24244d7adb90695623dcb4e2427f5ecf04a4363c40ead06aaaf8e"} err="failed to get container status \"8941d022aae24244d7adb90695623dcb4e2427f5ecf04a4363c40ead06aaaf8e\": rpc error: code = NotFound desc = could not find container \"8941d022aae24244d7adb90695623dcb4e2427f5ecf04a4363c40ead06aaaf8e\": container with ID starting with 8941d022aae24244d7adb90695623dcb4e2427f5ecf04a4363c40ead06aaaf8e not found: ID does not exist" Oct 30 19:20:48.809744 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:48.809620 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd525e12-1c0c-4ab5-91ac-9028a847cab6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd525e12-1c0c-4ab5-91ac-9028a847cab6-openshift-service-ca\") pod \"cd525e12-1c0c-4ab5-91ac-9028a847cab6\" (UID: \"cd525e12-1c0c-4ab5-91ac-9028a847cab6\") " Oct 30 19:20:48.810068 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:48.810028 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd525e12-1c0c-4ab5-91ac-9028a847cab6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd525e12-1c0c-4ab5-91ac-9028a847cab6-metrics-certs\") pod \"cd525e12-1c0c-4ab5-91ac-9028a847cab6\" (UID: \"cd525e12-1c0c-4ab5-91ac-9028a847cab6\") " Oct 30 19:20:48.810328 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:48.810279 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd525e12-1c0c-4ab5-91ac-9028a847cab6-kube-api-access-h2276\" (UniqueName: \"kubernetes.io/projected/cd525e12-1c0c-4ab5-91ac-9028a847cab6-kube-api-access-h2276\") pod \"cd525e12-1c0c-4ab5-91ac-9028a847cab6\" (UID: \"cd525e12-1c0c-4ab5-91ac-9028a847cab6\") " Oct 30 19:20:48.810808 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:48.810745 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd525e12-1c0c-4ab5-91ac-9028a847cab6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd525e12-1c0c-4ab5-91ac-9028a847cab6-maas-api-tls\") pod \"cd525e12-1c0c-4ab5-91ac-9028a847cab6\" (UID: \"cd525e12-1c0c-4ab5-91ac-9028a847cab6\") " Oct 30 19:20:48.825077 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:48.825022 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd525e12-1c0c-4ab5-91ac-9028a847cab6-maas-api-tls" pod "cd525e12-1c0c-4ab5-91ac-9028a847cab6" (UID: "cd525e12-1c0c-4ab5-91ac-9028a847cab6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:20:48.826479 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:48.826451 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd525e12-1c0c-4ab5-91ac-9028a847cab6-openshift-service-ca" pod "cd525e12-1c0c-4ab5-91ac-9028a847cab6" (UID: "cd525e12-1c0c-4ab5-91ac-9028a847cab6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:20:48.831765 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:48.831732 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd525e12-1c0c-4ab5-91ac-9028a847cab6-metrics-certs" pod "cd525e12-1c0c-4ab5-91ac-9028a847cab6" (UID: "cd525e12-1c0c-4ab5-91ac-9028a847cab6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:20:48.837221 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:48.836543 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd525e12-1c0c-4ab5-91ac-9028a847cab6-kube-api-access-h2276" pod "cd525e12-1c0c-4ab5-91ac-9028a847cab6" (UID: "cd525e12-1c0c-4ab5-91ac-9028a847cab6"). InnerVolumeSpecName "kube-api-access-h2276". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:20:48.854291 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:48.854018 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:48.914414 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:48.914378 6960 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd525e12-1c0c-4ab5-91ac-9028a847cab6-maas-api-tls\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:48.914414 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:48.914413 6960 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd525e12-1c0c-4ab5-91ac-9028a847cab6-openshift-service-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:48.914615 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:48.914430 6960 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd525e12-1c0c-4ab5-91ac-9028a847cab6-metrics-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:48.914615 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:48.914444 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2276\" (UniqueName: \"kubernetes.io/projected/cd525e12-1c0c-4ab5-91ac-9028a847cab6-kube-api-access-h2276\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:49.104765 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:49.104687 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-7c6027-7b8d4f4fb9-mpf4x"] Oct 30 19:20:49.116499 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:49.116470 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-7c6027-7b8d4f4fb9-mpf4x"] Oct 30 19:20:49.210113 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:49.210077 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:49.805997 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:49.805636 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:50.594756 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:50.594724 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd525e12-1c0c-4ab5-91ac-9028a847cab6" path="/var/lib/kubelet/pods/cd525e12-1c0c-4ab5-91ac-9028a847cab6/volumes" Oct 30 19:20:50.755820 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:50.755781 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:51.694191 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:51.689785 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:52.451175 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:52.451124 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-784fbf-6cdcf7dbc8-tq592"] Oct 30 19:20:52.451750 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:52.451721 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-784fbf-6cdcf7dbc8-tq592" podUID="c610dae0-c986-4c71-b430-9e1e7bd73e9b" containerName="maas-api" containerID="cri-o://a15ee0974b3f7189c18e196e6d97b15c4dec3cc481a3da02ccd10e173a5ad673" gracePeriod=30 Oct 30 19:20:52.489921 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:52.489889 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:52.637661 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:52.637622 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:52.922114 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:52.910966 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:53.039167 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:53.029000 6960 generic.go:356] "Generic (PLEG): container finished" podID="c610dae0-c986-4c71-b430-9e1e7bd73e9b" containerID="a15ee0974b3f7189c18e196e6d97b15c4dec3cc481a3da02ccd10e173a5ad673" exitCode=0 Oct 30 19:20:53.039167 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:53.029325 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-784fbf-6cdcf7dbc8-tq592" event={"ID":"c610dae0-c986-4c71-b430-9e1e7bd73e9b","Type":"ContainerDied","Data":"a15ee0974b3f7189c18e196e6d97b15c4dec3cc481a3da02ccd10e173a5ad673"} Oct 30 19:20:53.093176 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:53.092793 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-784fbf-6cdcf7dbc8-tq592" Oct 30 19:20:53.093999 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:53.093933 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:53.205228 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:53.205124 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:53.240803 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:53.240762 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c610dae0-c986-4c71-b430-9e1e7bd73e9b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c610dae0-c986-4c71-b430-9e1e7bd73e9b-openshift-service-ca\") pod \"c610dae0-c986-4c71-b430-9e1e7bd73e9b\" (UID: \"c610dae0-c986-4c71-b430-9e1e7bd73e9b\") " Oct 30 19:20:53.240969 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:53.240906 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c610dae0-c986-4c71-b430-9e1e7bd73e9b-kube-api-access-f8bg4\" (UniqueName: \"kubernetes.io/projected/c610dae0-c986-4c71-b430-9e1e7bd73e9b-kube-api-access-f8bg4\") pod \"c610dae0-c986-4c71-b430-9e1e7bd73e9b\" (UID: \"c610dae0-c986-4c71-b430-9e1e7bd73e9b\") " Oct 30 19:20:53.240969 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:53.240938 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c610dae0-c986-4c71-b430-9e1e7bd73e9b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c610dae0-c986-4c71-b430-9e1e7bd73e9b-maas-api-tls\") pod \"c610dae0-c986-4c71-b430-9e1e7bd73e9b\" (UID: \"c610dae0-c986-4c71-b430-9e1e7bd73e9b\") " Oct 30 19:20:53.241110 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:53.241042 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c610dae0-c986-4c71-b430-9e1e7bd73e9b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c610dae0-c986-4c71-b430-9e1e7bd73e9b-metrics-certs\") pod \"c610dae0-c986-4c71-b430-9e1e7bd73e9b\" (UID: \"c610dae0-c986-4c71-b430-9e1e7bd73e9b\") " Oct 30 19:20:53.241222 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:53.241184 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c610dae0-c986-4c71-b430-9e1e7bd73e9b-openshift-service-ca" pod "c610dae0-c986-4c71-b430-9e1e7bd73e9b" (UID: "c610dae0-c986-4c71-b430-9e1e7bd73e9b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:20:53.241599 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:53.241572 6960 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c610dae0-c986-4c71-b430-9e1e7bd73e9b-openshift-service-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:53.244079 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:53.244051 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c610dae0-c986-4c71-b430-9e1e7bd73e9b-maas-api-tls" pod "c610dae0-c986-4c71-b430-9e1e7bd73e9b" (UID: "c610dae0-c986-4c71-b430-9e1e7bd73e9b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:20:53.244079 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:53.244088 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c610dae0-c986-4c71-b430-9e1e7bd73e9b-metrics-certs" pod "c610dae0-c986-4c71-b430-9e1e7bd73e9b" (UID: "c610dae0-c986-4c71-b430-9e1e7bd73e9b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:20:53.246520 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:53.246491 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c610dae0-c986-4c71-b430-9e1e7bd73e9b-kube-api-access-f8bg4" pod "c610dae0-c986-4c71-b430-9e1e7bd73e9b" (UID: "c610dae0-c986-4c71-b430-9e1e7bd73e9b"). InnerVolumeSpecName "kube-api-access-f8bg4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:20:53.343654 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:53.343616 6960 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c610dae0-c986-4c71-b430-9e1e7bd73e9b-metrics-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:53.343654 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:53.343656 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8bg4\" (UniqueName: \"kubernetes.io/projected/c610dae0-c986-4c71-b430-9e1e7bd73e9b-kube-api-access-f8bg4\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:53.343941 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:53.343674 6960 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c610dae0-c986-4c71-b430-9e1e7bd73e9b-maas-api-tls\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:20:53.652827 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:53.652798 6960 scope.go:122] "RemoveContainer" containerID="a15ee0974b3f7189c18e196e6d97b15c4dec3cc481a3da02ccd10e173a5ad673" Oct 30 19:20:53.800199 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:53.800129 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:53.933534 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:20:53.933451 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 19:20:54.036191 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:54.036129 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-784fbf-6cdcf7dbc8-tq592" event={"ID":"c610dae0-c986-4c71-b430-9e1e7bd73e9b","Type":"ContainerDied","Data":"39770f40e993e93d9c4726d815570c6f01efb8c0a8f4c5dc1428d23ec8fa563f"} Oct 30 19:20:54.036370 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:54.036234 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-784fbf-6cdcf7dbc8-tq592" Oct 30 19:20:54.076231 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:54.076193 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-784fbf-6cdcf7dbc8-tq592"] Oct 30 19:20:54.086837 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:54.086811 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-784fbf-6cdcf7dbc8-tq592"] Oct 30 19:20:54.347106 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:54.347068 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:54.594918 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:54.593977 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c610dae0-c986-4c71-b430-9e1e7bd73e9b" path="/var/lib/kubelet/pods/c610dae0-c986-4c71-b430-9e1e7bd73e9b/volumes" Oct 30 19:20:54.610572 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:54.610501 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:55.319697 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:55.319669 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:56.119282 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:56.119249 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:56.961651 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:56.961616 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:57.129609 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:57.129577 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:57.299521 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:57.299442 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-pzmg6"] Oct 30 19:20:57.299718 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:57.299693 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-pzmg6" podUID="50c12729-1387-436a-bee0-9eb3316db708" containerName="istio-proxy" containerID="cri-o://25e863cbfc452efebbfa326f7074d5a7e3e3bcb51c9a80090e8d261dc45d26d3" gracePeriod=30 Oct 30 19:20:57.369463 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:57.369430 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:58.280039 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:58.280002 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:59.245160 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:59.245110 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:20:59.960916 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:59.960864 6960 patch_prober.go:29] interesting pod/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-pzmg6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:20:59.962065 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:20:59.962029 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-pzmg6" podUID="50c12729-1387-436a-bee0-9eb3316db708" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:21:00.079610 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:00.079576 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:01.222154 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:01.222103 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:01.699440 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:01.699407 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:01.845190 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:01.845155 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:02.056528 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:02.056498 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-kmw8d"] Oct 30 19:21:02.056735 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:02.056711 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-kmw8d" podUID="7b416cfb-588b-4dde-bd77-9d5fca225a87" containerName="istio-proxy" containerID="cri-o://36681657a405f967bda0f6142f8f4a3f6158ffc703f0dd4ee1ad969b96fb4877" gracePeriod=30 Oct 30 19:21:02.080214 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:02.080169 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:02.224304 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:02.224242 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:02.589274 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:02.589207 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-pzmg6" Oct 30 19:21:02.608160 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:02.608113 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:02.699833 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:02.699803 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50c12729-1387-436a-bee0-9eb3316db708-istio-token\" (UniqueName: \"kubernetes.io/projected/50c12729-1387-436a-bee0-9eb3316db708-istio-token\") pod \"50c12729-1387-436a-bee0-9eb3316db708\" (UID: \"50c12729-1387-436a-bee0-9eb3316db708\") " Oct 30 19:21:02.700034 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:02.699880 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50c12729-1387-436a-bee0-9eb3316db708-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50c12729-1387-436a-bee0-9eb3316db708-workload-certs\") pod \"50c12729-1387-436a-bee0-9eb3316db708\" (UID: \"50c12729-1387-436a-bee0-9eb3316db708\") " Oct 30 19:21:02.700034 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:02.699953 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50c12729-1387-436a-bee0-9eb3316db708-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50c12729-1387-436a-bee0-9eb3316db708-workload-socket\") pod \"50c12729-1387-436a-bee0-9eb3316db708\" (UID: \"50c12729-1387-436a-bee0-9eb3316db708\") " Oct 30 19:21:02.700034 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:02.700025 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50c12729-1387-436a-bee0-9eb3316db708-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50c12729-1387-436a-bee0-9eb3316db708-istio-envoy\") pod \"50c12729-1387-436a-bee0-9eb3316db708\" (UID: \"50c12729-1387-436a-bee0-9eb3316db708\") " Oct 30 19:21:02.700239 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:02.700074 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50c12729-1387-436a-bee0-9eb3316db708-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50c12729-1387-436a-bee0-9eb3316db708-credential-socket\") pod \"50c12729-1387-436a-bee0-9eb3316db708\" (UID: \"50c12729-1387-436a-bee0-9eb3316db708\") " Oct 30 19:21:02.700239 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:02.700105 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50c12729-1387-436a-bee0-9eb3316db708-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50c12729-1387-436a-bee0-9eb3316db708-istio-data\") pod \"50c12729-1387-436a-bee0-9eb3316db708\" (UID: \"50c12729-1387-436a-bee0-9eb3316db708\") " Oct 30 19:21:02.700239 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:02.700216 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50c12729-1387-436a-bee0-9eb3316db708-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50c12729-1387-436a-bee0-9eb3316db708-istiod-ca-cert\") pod \"50c12729-1387-436a-bee0-9eb3316db708\" (UID: \"50c12729-1387-436a-bee0-9eb3316db708\") " Oct 30 19:21:02.700239 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:02.700222 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50c12729-1387-436a-bee0-9eb3316db708-workload-socket" pod "50c12729-1387-436a-bee0-9eb3316db708" (UID: "50c12729-1387-436a-bee0-9eb3316db708"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:21:02.700456 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:02.700242 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50c12729-1387-436a-bee0-9eb3316db708-workload-certs" pod "50c12729-1387-436a-bee0-9eb3316db708" (UID: "50c12729-1387-436a-bee0-9eb3316db708"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:21:02.700456 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:02.700273 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/50c12729-1387-436a-bee0-9eb3316db708-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50c12729-1387-436a-bee0-9eb3316db708-istio-podinfo\") pod \"50c12729-1387-436a-bee0-9eb3316db708\" (UID: \"50c12729-1387-436a-bee0-9eb3316db708\") " Oct 30 19:21:02.700456 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:02.700316 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50c12729-1387-436a-bee0-9eb3316db708-credential-socket" pod "50c12729-1387-436a-bee0-9eb3316db708" (UID: "50c12729-1387-436a-bee0-9eb3316db708"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:21:02.700456 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:02.700409 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50c12729-1387-436a-bee0-9eb3316db708-istio-data" pod "50c12729-1387-436a-bee0-9eb3316db708" (UID: "50c12729-1387-436a-bee0-9eb3316db708"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:21:02.700456 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:02.700449 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50c12729-1387-436a-bee0-9eb3316db708-kube-api-access-vrpcg\" (UniqueName: \"kubernetes.io/projected/50c12729-1387-436a-bee0-9eb3316db708-kube-api-access-vrpcg\") pod \"50c12729-1387-436a-bee0-9eb3316db708\" (UID: \"50c12729-1387-436a-bee0-9eb3316db708\") " Oct 30 19:21:02.700739 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:02.700589 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50c12729-1387-436a-bee0-9eb3316db708-istiod-ca-cert" pod "50c12729-1387-436a-bee0-9eb3316db708" (UID: "50c12729-1387-436a-bee0-9eb3316db708"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:21:02.701307 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:02.701281 6960 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50c12729-1387-436a-bee0-9eb3316db708-istiod-ca-cert\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:21:02.701453 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:02.701309 6960 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50c12729-1387-436a-bee0-9eb3316db708-workload-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:21:02.701453 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:02.701327 6960 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50c12729-1387-436a-bee0-9eb3316db708-workload-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:21:02.701453 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:02.701344 6960 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50c12729-1387-436a-bee0-9eb3316db708-credential-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:21:02.701453 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:02.701360 6960 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50c12729-1387-436a-bee0-9eb3316db708-istio-data\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:21:02.702551 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:02.702522 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50c12729-1387-436a-bee0-9eb3316db708-istio-envoy" pod "50c12729-1387-436a-bee0-9eb3316db708" (UID: "50c12729-1387-436a-bee0-9eb3316db708"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:21:02.702654 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:02.702628 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50c12729-1387-436a-bee0-9eb3316db708-istio-token" pod "50c12729-1387-436a-bee0-9eb3316db708" (UID: "50c12729-1387-436a-bee0-9eb3316db708"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:21:02.702725 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:02.702709 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/50c12729-1387-436a-bee0-9eb3316db708-istio-podinfo" pod "50c12729-1387-436a-bee0-9eb3316db708" (UID: "50c12729-1387-436a-bee0-9eb3316db708"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:21:02.703123 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:02.703104 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50c12729-1387-436a-bee0-9eb3316db708-kube-api-access-vrpcg" pod "50c12729-1387-436a-bee0-9eb3316db708" (UID: "50c12729-1387-436a-bee0-9eb3316db708"). InnerVolumeSpecName "kube-api-access-vrpcg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:21:02.802920 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:02.802886 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrpcg\" (UniqueName: \"kubernetes.io/projected/50c12729-1387-436a-bee0-9eb3316db708-kube-api-access-vrpcg\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:21:02.802920 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:02.802919 6960 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50c12729-1387-436a-bee0-9eb3316db708-istio-token\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:21:02.803166 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:02.802933 6960 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50c12729-1387-436a-bee0-9eb3316db708-istio-envoy\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:21:02.803166 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:02.802947 6960 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50c12729-1387-436a-bee0-9eb3316db708-istio-podinfo\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:21:03.103729 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:03.103693 6960 generic.go:356] "Generic (PLEG): container finished" podID="50c12729-1387-436a-bee0-9eb3316db708" containerID="25e863cbfc452efebbfa326f7074d5a7e3e3bcb51c9a80090e8d261dc45d26d3" exitCode=0 Oct 30 19:21:03.103968 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:03.103746 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-pzmg6" Oct 30 19:21:03.103968 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:03.103780 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-pzmg6" event={"ID":"50c12729-1387-436a-bee0-9eb3316db708","Type":"ContainerDied","Data":"25e863cbfc452efebbfa326f7074d5a7e3e3bcb51c9a80090e8d261dc45d26d3"} Oct 30 19:21:03.103968 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:03.103837 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-pzmg6" event={"ID":"50c12729-1387-436a-bee0-9eb3316db708","Type":"ContainerDied","Data":"b8887ac9dbd301fd7badfb0721c9ed2d6dc611b2b7ccf7d05aab7c4b823f5146"} Oct 30 19:21:03.103968 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:03.103862 6960 scope.go:122] "RemoveContainer" containerID="25e863cbfc452efebbfa326f7074d5a7e3e3bcb51c9a80090e8d261dc45d26d3" Oct 30 19:21:03.116507 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:03.116470 6960 scope.go:122] "RemoveContainer" containerID="25e863cbfc452efebbfa326f7074d5a7e3e3bcb51c9a80090e8d261dc45d26d3" Oct 30 19:21:03.118057 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:21:03.117288 6960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25e863cbfc452efebbfa326f7074d5a7e3e3bcb51c9a80090e8d261dc45d26d3\": container with ID starting with 25e863cbfc452efebbfa326f7074d5a7e3e3bcb51c9a80090e8d261dc45d26d3 not found: ID does not exist" containerID="25e863cbfc452efebbfa326f7074d5a7e3e3bcb51c9a80090e8d261dc45d26d3" Oct 30 19:21:03.118057 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:03.117331 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25e863cbfc452efebbfa326f7074d5a7e3e3bcb51c9a80090e8d261dc45d26d3"} err="failed to get container status \"25e863cbfc452efebbfa326f7074d5a7e3e3bcb51c9a80090e8d261dc45d26d3\": rpc error: code = NotFound desc = could not find container \"25e863cbfc452efebbfa326f7074d5a7e3e3bcb51c9a80090e8d261dc45d26d3\": container with ID starting with 25e863cbfc452efebbfa326f7074d5a7e3e3bcb51c9a80090e8d261dc45d26d3 not found: ID does not exist" Oct 30 19:21:03.131110 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:03.131078 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-pzmg6"] Oct 30 19:21:03.139941 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:03.139911 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-7c6027-openshift-default-69dfbcbb9c-pzmg6"] Oct 30 19:21:03.445300 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:03.445226 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-03362d-655b6cd7f8-ndl7x"] Oct 30 19:21:03.471450 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:03.471415 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-03362d-5df77d4775-tn9s2"] Oct 30 19:21:03.535926 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:03.535890 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:03.651637 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:03.651603 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:04.426432 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:04.426387 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:04.594273 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:04.594238 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50c12729-1387-436a-bee0-9eb3316db708" path="/var/lib/kubelet/pods/50c12729-1387-436a-bee0-9eb3316db708/volumes" Oct 30 19:21:05.355886 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:05.355848 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:06.158886 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:06.158832 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:06.749768 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:06.749732 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:06.901598 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:06.901564 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:07.383602 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:07.383576 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-kmw8d" Oct 30 19:21:07.505658 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:07.505627 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b416cfb-588b-4dde-bd77-9d5fca225a87-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b416cfb-588b-4dde-bd77-9d5fca225a87-istiod-ca-cert\") pod \"7b416cfb-588b-4dde-bd77-9d5fca225a87\" (UID: \"7b416cfb-588b-4dde-bd77-9d5fca225a87\") " Oct 30 19:21:07.505856 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:07.505685 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b416cfb-588b-4dde-bd77-9d5fca225a87-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b416cfb-588b-4dde-bd77-9d5fca225a87-workload-certs\") pod \"7b416cfb-588b-4dde-bd77-9d5fca225a87\" (UID: \"7b416cfb-588b-4dde-bd77-9d5fca225a87\") " Oct 30 19:21:07.505856 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:07.505804 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b416cfb-588b-4dde-bd77-9d5fca225a87-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b416cfb-588b-4dde-bd77-9d5fca225a87-workload-socket\") pod \"7b416cfb-588b-4dde-bd77-9d5fca225a87\" (UID: \"7b416cfb-588b-4dde-bd77-9d5fca225a87\") " Oct 30 19:21:07.505978 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:07.505877 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b416cfb-588b-4dde-bd77-9d5fca225a87-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b416cfb-588b-4dde-bd77-9d5fca225a87-istio-data\") pod \"7b416cfb-588b-4dde-bd77-9d5fca225a87\" (UID: \"7b416cfb-588b-4dde-bd77-9d5fca225a87\") " Oct 30 19:21:07.505978 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:07.505965 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b416cfb-588b-4dde-bd77-9d5fca225a87-istio-token\" (UniqueName: \"kubernetes.io/projected/7b416cfb-588b-4dde-bd77-9d5fca225a87-istio-token\") pod \"7b416cfb-588b-4dde-bd77-9d5fca225a87\" (UID: \"7b416cfb-588b-4dde-bd77-9d5fca225a87\") " Oct 30 19:21:07.506086 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:07.505972 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b416cfb-588b-4dde-bd77-9d5fca225a87-workload-certs" pod "7b416cfb-588b-4dde-bd77-9d5fca225a87" (UID: "7b416cfb-588b-4dde-bd77-9d5fca225a87"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:21:07.506086 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:07.506014 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7b416cfb-588b-4dde-bd77-9d5fca225a87-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b416cfb-588b-4dde-bd77-9d5fca225a87-istio-podinfo\") pod \"7b416cfb-588b-4dde-bd77-9d5fca225a87\" (UID: \"7b416cfb-588b-4dde-bd77-9d5fca225a87\") " Oct 30 19:21:07.506242 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:07.506084 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b416cfb-588b-4dde-bd77-9d5fca225a87-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b416cfb-588b-4dde-bd77-9d5fca225a87-credential-socket\") pod \"7b416cfb-588b-4dde-bd77-9d5fca225a87\" (UID: \"7b416cfb-588b-4dde-bd77-9d5fca225a87\") " Oct 30 19:21:07.506242 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:07.506095 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b416cfb-588b-4dde-bd77-9d5fca225a87-workload-socket" pod "7b416cfb-588b-4dde-bd77-9d5fca225a87" (UID: "7b416cfb-588b-4dde-bd77-9d5fca225a87"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:21:07.506242 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:07.506134 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b416cfb-588b-4dde-bd77-9d5fca225a87-kube-api-access-hrfl8\" (UniqueName: \"kubernetes.io/projected/7b416cfb-588b-4dde-bd77-9d5fca225a87-kube-api-access-hrfl8\") pod \"7b416cfb-588b-4dde-bd77-9d5fca225a87\" (UID: \"7b416cfb-588b-4dde-bd77-9d5fca225a87\") " Oct 30 19:21:07.506242 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:07.506188 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b416cfb-588b-4dde-bd77-9d5fca225a87-istiod-ca-cert" pod "7b416cfb-588b-4dde-bd77-9d5fca225a87" (UID: "7b416cfb-588b-4dde-bd77-9d5fca225a87"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:21:07.506242 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:07.506202 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b416cfb-588b-4dde-bd77-9d5fca225a87-istio-data" pod "7b416cfb-588b-4dde-bd77-9d5fca225a87" (UID: "7b416cfb-588b-4dde-bd77-9d5fca225a87"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:21:07.506242 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:07.506241 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b416cfb-588b-4dde-bd77-9d5fca225a87-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b416cfb-588b-4dde-bd77-9d5fca225a87-istio-envoy\") pod \"7b416cfb-588b-4dde-bd77-9d5fca225a87\" (UID: \"7b416cfb-588b-4dde-bd77-9d5fca225a87\") " Oct 30 19:21:07.506533 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:07.506328 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b416cfb-588b-4dde-bd77-9d5fca225a87-credential-socket" pod "7b416cfb-588b-4dde-bd77-9d5fca225a87" (UID: "7b416cfb-588b-4dde-bd77-9d5fca225a87"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:21:07.507024 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:07.506890 6960 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b416cfb-588b-4dde-bd77-9d5fca225a87-istiod-ca-cert\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:21:07.507024 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:07.506916 6960 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b416cfb-588b-4dde-bd77-9d5fca225a87-workload-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:21:07.507024 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:07.506930 6960 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b416cfb-588b-4dde-bd77-9d5fca225a87-workload-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:21:07.507024 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:07.506944 6960 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b416cfb-588b-4dde-bd77-9d5fca225a87-istio-data\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:21:07.507024 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:07.506957 6960 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b416cfb-588b-4dde-bd77-9d5fca225a87-credential-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:21:07.508946 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:07.508925 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b416cfb-588b-4dde-bd77-9d5fca225a87-kube-api-access-hrfl8" pod "7b416cfb-588b-4dde-bd77-9d5fca225a87" (UID: "7b416cfb-588b-4dde-bd77-9d5fca225a87"). InnerVolumeSpecName "kube-api-access-hrfl8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:21:07.509084 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:07.509068 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b416cfb-588b-4dde-bd77-9d5fca225a87-istio-envoy" pod "7b416cfb-588b-4dde-bd77-9d5fca225a87" (UID: "7b416cfb-588b-4dde-bd77-9d5fca225a87"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:21:07.509464 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:07.509440 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7b416cfb-588b-4dde-bd77-9d5fca225a87-istio-podinfo" pod "7b416cfb-588b-4dde-bd77-9d5fca225a87" (UID: "7b416cfb-588b-4dde-bd77-9d5fca225a87"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:21:07.509966 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:07.509945 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b416cfb-588b-4dde-bd77-9d5fca225a87-istio-token" pod "7b416cfb-588b-4dde-bd77-9d5fca225a87" (UID: "7b416cfb-588b-4dde-bd77-9d5fca225a87"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:21:07.608375 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:07.608340 6960 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b416cfb-588b-4dde-bd77-9d5fca225a87-istio-token\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:21:07.608375 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:07.608373 6960 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b416cfb-588b-4dde-bd77-9d5fca225a87-istio-podinfo\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:21:07.608578 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:07.608389 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrfl8\" (UniqueName: \"kubernetes.io/projected/7b416cfb-588b-4dde-bd77-9d5fca225a87-kube-api-access-hrfl8\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:21:07.608578 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:07.608403 6960 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b416cfb-588b-4dde-bd77-9d5fca225a87-istio-envoy\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:21:07.648671 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:07.648130 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-aa6736/distinct-w1-aa6736-kserve-b86d7667b-smcbz"] Oct 30 19:21:07.659040 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:07.656775 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-aa6736/premium-w1-aa6736-kserve-6f986b66f7-nkfrg"] Oct 30 19:21:07.665301 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:07.665254 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-aa6736/simulator-w1-aa6736-kserve-76985dcd5b-njvnn"] Oct 30 19:21:07.746829 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:07.746569 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:07.883015 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:07.880913 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:08.169454 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:08.168733 6960 generic.go:356] "Generic (PLEG): container finished" podID="7b416cfb-588b-4dde-bd77-9d5fca225a87" containerID="36681657a405f967bda0f6142f8f4a3f6158ffc703f0dd4ee1ad969b96fb4877" exitCode=0 Oct 30 19:21:08.169454 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:08.168781 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-kmw8d" Oct 30 19:21:08.169454 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:08.168807 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-kmw8d" event={"ID":"7b416cfb-588b-4dde-bd77-9d5fca225a87","Type":"ContainerDied","Data":"36681657a405f967bda0f6142f8f4a3f6158ffc703f0dd4ee1ad969b96fb4877"} Oct 30 19:21:08.169454 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:08.168867 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-kmw8d" event={"ID":"7b416cfb-588b-4dde-bd77-9d5fca225a87","Type":"ContainerDied","Data":"50812907dadcc5325cdfa094fd41eb30917ed82a3f3d6bf90392279e81bcc8cd"} Oct 30 19:21:08.169454 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:08.168897 6960 scope.go:122] "RemoveContainer" containerID="36681657a405f967bda0f6142f8f4a3f6158ffc703f0dd4ee1ad969b96fb4877" Oct 30 19:21:08.187369 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:08.187348 6960 scope.go:122] "RemoveContainer" containerID="36681657a405f967bda0f6142f8f4a3f6158ffc703f0dd4ee1ad969b96fb4877" Oct 30 19:21:08.187695 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:21:08.187679 6960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36681657a405f967bda0f6142f8f4a3f6158ffc703f0dd4ee1ad969b96fb4877\": container with ID starting with 36681657a405f967bda0f6142f8f4a3f6158ffc703f0dd4ee1ad969b96fb4877 not found: ID does not exist" containerID="36681657a405f967bda0f6142f8f4a3f6158ffc703f0dd4ee1ad969b96fb4877" Oct 30 19:21:08.187754 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:08.187707 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36681657a405f967bda0f6142f8f4a3f6158ffc703f0dd4ee1ad969b96fb4877"} err="failed to get container status \"36681657a405f967bda0f6142f8f4a3f6158ffc703f0dd4ee1ad969b96fb4877\": rpc error: code = NotFound desc = could not find container \"36681657a405f967bda0f6142f8f4a3f6158ffc703f0dd4ee1ad969b96fb4877\": container with ID starting with 36681657a405f967bda0f6142f8f4a3f6158ffc703f0dd4ee1ad969b96fb4877 not found: ID does not exist" Oct 30 19:21:08.198035 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:08.198012 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-kmw8d"] Oct 30 19:21:08.204940 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:08.204917 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:08.205574 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:08.205551 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-784fbf-openshift-default-89c8796b9-kmw8d"] Oct 30 19:21:08.374511 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:08.374439 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:08.447275 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:08.446231 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-03362d-655b6cd7f8-ndl7x" podUID="b5a06a95-d251-46b6-889c-7e7d5e2ca925" containerName="payload-processing" containerID="cri-o://2470337503e5e11c7b374c3739b0d98a86e584f1f3db5fec12207fc2f6d2daca" gracePeriod=25 Oct 30 19:21:08.473167 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:08.472392 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-03362d-5df77d4775-tn9s2" podUID="c8690e67-62d5-4a1b-8632-a23a7d2f2843" containerName="payload-pre-processing" containerID="cri-o://2de73ed0cef9082202870f240fb4c702b838bc4a07eb49a8f74f61a1eb528621" gracePeriod=25 Oct 30 19:21:08.594267 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:08.594232 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b416cfb-588b-4dde-bd77-9d5fca225a87" path="/var/lib/kubelet/pods/7b416cfb-588b-4dde-bd77-9d5fca225a87/volumes" Oct 30 19:21:08.819403 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:08.819363 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:09.659379 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:09.659349 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:10.520041 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:10.520008 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:11.335842 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:11.335807 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:11.359413 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:11.359323 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-03362d-655b6cd7f8-ndl7x" podUID="b5a06a95-d251-46b6-889c-7e7d5e2ca925" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:21:11.360007 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:11.359963 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-03362d-5df77d4775-tn9s2" podUID="c8690e67-62d5-4a1b-8632-a23a7d2f2843" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:21:12.198684 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:12.198651 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:12.985164 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:12.978717 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:13.044100 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:13.044065 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-03362d-6fdbdf99fb-72dp5"] Oct 30 19:21:13.044379 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:13.044346 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-03362d-6fdbdf99fb-72dp5" podUID="efd98aa3-a350-4bf9-82ec-34f7b7917adc" containerName="maas-api" containerID="cri-o://fb1bc614e60821470b40ad5260ca11e70a89f7cfce208380e89642a08aeaeb70" gracePeriod=30 Oct 30 19:21:13.212884 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:13.212831 6960 generic.go:356] "Generic (PLEG): container finished" podID="efd98aa3-a350-4bf9-82ec-34f7b7917adc" containerID="fb1bc614e60821470b40ad5260ca11e70a89f7cfce208380e89642a08aeaeb70" exitCode=0 Oct 30 19:21:13.213055 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:13.212914 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-03362d-6fdbdf99fb-72dp5" event={"ID":"efd98aa3-a350-4bf9-82ec-34f7b7917adc","Type":"ContainerDied","Data":"fb1bc614e60821470b40ad5260ca11e70a89f7cfce208380e89642a08aeaeb70"} Oct 30 19:21:13.223540 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:13.223508 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:13.403215 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:13.394061 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:13.425623 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:13.420529 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-03362d-6fdbdf99fb-72dp5" Oct 30 19:21:13.549272 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:13.549236 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/efd98aa3-a350-4bf9-82ec-34f7b7917adc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/efd98aa3-a350-4bf9-82ec-34f7b7917adc-metrics-certs\") pod \"efd98aa3-a350-4bf9-82ec-34f7b7917adc\" (UID: \"efd98aa3-a350-4bf9-82ec-34f7b7917adc\") " Oct 30 19:21:13.549443 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:13.549406 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/efd98aa3-a350-4bf9-82ec-34f7b7917adc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/efd98aa3-a350-4bf9-82ec-34f7b7917adc-maas-api-tls\") pod \"efd98aa3-a350-4bf9-82ec-34f7b7917adc\" (UID: \"efd98aa3-a350-4bf9-82ec-34f7b7917adc\") " Oct 30 19:21:13.549518 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:13.549502 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efd98aa3-a350-4bf9-82ec-34f7b7917adc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/efd98aa3-a350-4bf9-82ec-34f7b7917adc-openshift-service-ca\") pod \"efd98aa3-a350-4bf9-82ec-34f7b7917adc\" (UID: \"efd98aa3-a350-4bf9-82ec-34f7b7917adc\") " Oct 30 19:21:13.549562 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:13.549547 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efd98aa3-a350-4bf9-82ec-34f7b7917adc-kube-api-access-cbv87\" (UniqueName: \"kubernetes.io/projected/efd98aa3-a350-4bf9-82ec-34f7b7917adc-kube-api-access-cbv87\") pod \"efd98aa3-a350-4bf9-82ec-34f7b7917adc\" (UID: \"efd98aa3-a350-4bf9-82ec-34f7b7917adc\") " Oct 30 19:21:13.549910 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:13.549885 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efd98aa3-a350-4bf9-82ec-34f7b7917adc-openshift-service-ca" pod "efd98aa3-a350-4bf9-82ec-34f7b7917adc" (UID: "efd98aa3-a350-4bf9-82ec-34f7b7917adc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:21:13.550110 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:13.550095 6960 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/efd98aa3-a350-4bf9-82ec-34f7b7917adc-openshift-service-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:21:13.551600 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:13.551575 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efd98aa3-a350-4bf9-82ec-34f7b7917adc-metrics-certs" pod "efd98aa3-a350-4bf9-82ec-34f7b7917adc" (UID: "efd98aa3-a350-4bf9-82ec-34f7b7917adc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:21:13.552226 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:13.552202 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efd98aa3-a350-4bf9-82ec-34f7b7917adc-maas-api-tls" pod "efd98aa3-a350-4bf9-82ec-34f7b7917adc" (UID: "efd98aa3-a350-4bf9-82ec-34f7b7917adc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:21:13.552292 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:13.552243 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efd98aa3-a350-4bf9-82ec-34f7b7917adc-kube-api-access-cbv87" pod "efd98aa3-a350-4bf9-82ec-34f7b7917adc" (UID: "efd98aa3-a350-4bf9-82ec-34f7b7917adc"). InnerVolumeSpecName "kube-api-access-cbv87". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:21:13.594895 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:13.594817 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:13.650846 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:13.650809 6960 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/efd98aa3-a350-4bf9-82ec-34f7b7917adc-maas-api-tls\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:21:13.650846 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:13.650846 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbv87\" (UniqueName: \"kubernetes.io/projected/efd98aa3-a350-4bf9-82ec-34f7b7917adc-kube-api-access-cbv87\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:21:13.651051 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:13.650859 6960 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/efd98aa3-a350-4bf9-82ec-34f7b7917adc-metrics-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:21:13.718606 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:13.718478 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:14.220855 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:14.220828 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-03362d-6fdbdf99fb-72dp5" Oct 30 19:21:14.221335 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:14.220857 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-03362d-6fdbdf99fb-72dp5" event={"ID":"efd98aa3-a350-4bf9-82ec-34f7b7917adc","Type":"ContainerDied","Data":"9465bee0a705addb914204a4b0e6ebabf16d076d41940d76b08d2cb06e2d54bf"} Oct 30 19:21:14.221335 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:14.220912 6960 scope.go:122] "RemoveContainer" containerID="fb1bc614e60821470b40ad5260ca11e70a89f7cfce208380e89642a08aeaeb70" Oct 30 19:21:14.272851 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:14.272822 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:14.287739 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:14.287715 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-03362d-6fdbdf99fb-72dp5"] Oct 30 19:21:14.291233 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:14.291212 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-03362d-6fdbdf99fb-72dp5"] Oct 30 19:21:14.594519 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:14.594484 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efd98aa3-a350-4bf9-82ec-34f7b7917adc" path="/var/lib/kubelet/pods/efd98aa3-a350-4bf9-82ec-34f7b7917adc/volumes" Oct 30 19:21:15.711815 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:15.709757 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:16.145894 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:16.145860 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:17.141106 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:17.141078 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:18.095752 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:18.095717 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:18.882492 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:18.882453 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:19.565642 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:19.565606 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:19.795900 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:19.795866 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:19.930745 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:19.930658 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:20.052200 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:20.052130 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:20.451779 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:20.451746 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:21.268105 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:21.268072 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:21.367540 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:21.367491 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-03362d-655b6cd7f8-ndl7x" podUID="b5a06a95-d251-46b6-889c-7e7d5e2ca925" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:21:21.370740 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:21.370702 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-03362d-5df77d4775-tn9s2" podUID="c8690e67-62d5-4a1b-8632-a23a7d2f2843" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:21:22.024627 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:22.024593 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:22.164998 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:22.164924 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:22.302908 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:22.302809 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:22.453366 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:22.453338 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:22.685632 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:22.685590 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-aa6736/distinct-w1-aa6736-kserve-b86d7667b-smcbz" podUID="58c23db7-11cb-4ce2-a67e-fe698e89c71d" containerName="main" containerID="cri-o://c5915f27d8d9f6e341312f052cc2e232ecd2170eeb1f9fc05a374f08aa593681" gracePeriod=45 Oct 30 19:21:22.689216 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:22.689181 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-aa6736/premium-w1-aa6736-kserve-6f986b66f7-nkfrg" podUID="84ca40c5-7033-48c4-bcec-e9ac7c034a4e" containerName="main" containerID="cri-o://3f5ce52a570bb1a7e8b84a47e71b27cbcb6454e8f194ce65a6e555ae7b5bb2b2" gracePeriod=45 Oct 30 19:21:22.703719 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:22.703682 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-aa6736/simulator-w1-aa6736-kserve-76985dcd5b-njvnn" podUID="943f7840-8d54-4845-b7ec-c0da72ba9c67" containerName="main" containerID="cri-o://af2a3147a3a36da06b44805485145efe0622412f4c5e6c9f94c0b755a6811f2b" gracePeriod=45 Oct 30 19:21:22.914903 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:22.914853 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:23.013426 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:23.013365 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-aa6736/distinct-w1-aa6736-kserve-b86d7667b-smcbz" Oct 30 19:21:23.022386 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:23.022360 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-aa6736/premium-w1-aa6736-kserve-6f986b66f7-nkfrg" Oct 30 19:21:23.046706 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:23.046671 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-aa6736/simulator-w1-aa6736-kserve-76985dcd5b-njvnn" Oct 30 19:21:23.149127 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:23.149037 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58c23db7-11cb-4ce2-a67e-fe698e89c71d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/58c23db7-11cb-4ce2-a67e-fe698e89c71d-dshm\") pod \"58c23db7-11cb-4ce2-a67e-fe698e89c71d\" (UID: \"58c23db7-11cb-4ce2-a67e-fe698e89c71d\") " Oct 30 19:21:23.149127 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:23.149103 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58c23db7-11cb-4ce2-a67e-fe698e89c71d-kube-api-access-hv4j5\" (UniqueName: \"kubernetes.io/projected/58c23db7-11cb-4ce2-a67e-fe698e89c71d-kube-api-access-hv4j5\") pod \"58c23db7-11cb-4ce2-a67e-fe698e89c71d\" (UID: \"58c23db7-11cb-4ce2-a67e-fe698e89c71d\") " Oct 30 19:21:23.149127 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:23.149127 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/943f7840-8d54-4845-b7ec-c0da72ba9c67-kube-api-access-x2xhz\" (UniqueName: \"kubernetes.io/projected/943f7840-8d54-4845-b7ec-c0da72ba9c67-kube-api-access-x2xhz\") pod \"943f7840-8d54-4845-b7ec-c0da72ba9c67\" (UID: \"943f7840-8d54-4845-b7ec-c0da72ba9c67\") " Oct 30 19:21:23.149475 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:23.149167 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58c23db7-11cb-4ce2-a67e-fe698e89c71d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58c23db7-11cb-4ce2-a67e-fe698e89c71d-model-cache\") pod \"58c23db7-11cb-4ce2-a67e-fe698e89c71d\" (UID: \"58c23db7-11cb-4ce2-a67e-fe698e89c71d\") " Oct 30 19:21:23.149475 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:23.149208 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84ca40c5-7033-48c4-bcec-e9ac7c034a4e-home\" (UniqueName: \"kubernetes.io/empty-dir/84ca40c5-7033-48c4-bcec-e9ac7c034a4e-home\") pod \"84ca40c5-7033-48c4-bcec-e9ac7c034a4e\" (UID: \"84ca40c5-7033-48c4-bcec-e9ac7c034a4e\") " Oct 30 19:21:23.149475 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:23.149241 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/943f7840-8d54-4845-b7ec-c0da72ba9c67-tls-certs\" (UniqueName: \"kubernetes.io/secret/943f7840-8d54-4845-b7ec-c0da72ba9c67-tls-certs\") pod \"943f7840-8d54-4845-b7ec-c0da72ba9c67\" (UID: \"943f7840-8d54-4845-b7ec-c0da72ba9c67\") " Oct 30 19:21:23.149475 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:23.149282 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84ca40c5-7033-48c4-bcec-e9ac7c034a4e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84ca40c5-7033-48c4-bcec-e9ac7c034a4e-tmp-dir\") pod \"84ca40c5-7033-48c4-bcec-e9ac7c034a4e\" (UID: \"84ca40c5-7033-48c4-bcec-e9ac7c034a4e\") " Oct 30 19:21:23.149475 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:23.149306 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84ca40c5-7033-48c4-bcec-e9ac7c034a4e-kube-api-access-k5xlc\" (UniqueName: \"kubernetes.io/projected/84ca40c5-7033-48c4-bcec-e9ac7c034a4e-kube-api-access-k5xlc\") pod \"84ca40c5-7033-48c4-bcec-e9ac7c034a4e\" (UID: \"84ca40c5-7033-48c4-bcec-e9ac7c034a4e\") " Oct 30 19:21:23.149475 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:23.149375 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58c23db7-11cb-4ce2-a67e-fe698e89c71d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58c23db7-11cb-4ce2-a67e-fe698e89c71d-tmp-dir\") pod \"58c23db7-11cb-4ce2-a67e-fe698e89c71d\" (UID: \"58c23db7-11cb-4ce2-a67e-fe698e89c71d\") " Oct 30 19:21:23.149475 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:23.149462 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84ca40c5-7033-48c4-bcec-e9ac7c034a4e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/84ca40c5-7033-48c4-bcec-e9ac7c034a4e-dshm\") pod \"84ca40c5-7033-48c4-bcec-e9ac7c034a4e\" (UID: \"84ca40c5-7033-48c4-bcec-e9ac7c034a4e\") " Oct 30 19:21:23.149793 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:23.149517 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84ca40c5-7033-48c4-bcec-e9ac7c034a4e-tls-certs\" (UniqueName: \"kubernetes.io/secret/84ca40c5-7033-48c4-bcec-e9ac7c034a4e-tls-certs\") pod \"84ca40c5-7033-48c4-bcec-e9ac7c034a4e\" (UID: \"84ca40c5-7033-48c4-bcec-e9ac7c034a4e\") " Oct 30 19:21:23.149793 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:23.149499 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84ca40c5-7033-48c4-bcec-e9ac7c034a4e-home" pod "84ca40c5-7033-48c4-bcec-e9ac7c034a4e" (UID: "84ca40c5-7033-48c4-bcec-e9ac7c034a4e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:21:23.149793 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:23.149553 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84ca40c5-7033-48c4-bcec-e9ac7c034a4e-tmp-dir" pod "84ca40c5-7033-48c4-bcec-e9ac7c034a4e" (UID: "84ca40c5-7033-48c4-bcec-e9ac7c034a4e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:21:23.149793 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:23.149566 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/943f7840-8d54-4845-b7ec-c0da72ba9c67-home\" (UniqueName: \"kubernetes.io/empty-dir/943f7840-8d54-4845-b7ec-c0da72ba9c67-home\") pod \"943f7840-8d54-4845-b7ec-c0da72ba9c67\" (UID: \"943f7840-8d54-4845-b7ec-c0da72ba9c67\") " Oct 30 19:21:23.149793 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:23.149673 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58c23db7-11cb-4ce2-a67e-fe698e89c71d-model-cache" pod "58c23db7-11cb-4ce2-a67e-fe698e89c71d" (UID: "58c23db7-11cb-4ce2-a67e-fe698e89c71d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:21:23.149793 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:23.149688 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/943f7840-8d54-4845-b7ec-c0da72ba9c67-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/943f7840-8d54-4845-b7ec-c0da72ba9c67-tmp-dir\") pod \"943f7840-8d54-4845-b7ec-c0da72ba9c67\" (UID: \"943f7840-8d54-4845-b7ec-c0da72ba9c67\") " Oct 30 19:21:23.149793 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:23.149745 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/943f7840-8d54-4845-b7ec-c0da72ba9c67-dshm\" (UniqueName: \"kubernetes.io/empty-dir/943f7840-8d54-4845-b7ec-c0da72ba9c67-dshm\") pod \"943f7840-8d54-4845-b7ec-c0da72ba9c67\" (UID: \"943f7840-8d54-4845-b7ec-c0da72ba9c67\") " Oct 30 19:21:23.149793 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:23.149793 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58c23db7-11cb-4ce2-a67e-fe698e89c71d-home\" (UniqueName: \"kubernetes.io/empty-dir/58c23db7-11cb-4ce2-a67e-fe698e89c71d-home\") pod \"58c23db7-11cb-4ce2-a67e-fe698e89c71d\" (UID: \"58c23db7-11cb-4ce2-a67e-fe698e89c71d\") " Oct 30 19:21:23.150409 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:23.149822 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58c23db7-11cb-4ce2-a67e-fe698e89c71d-tls-certs\" (UniqueName: \"kubernetes.io/secret/58c23db7-11cb-4ce2-a67e-fe698e89c71d-tls-certs\") pod \"58c23db7-11cb-4ce2-a67e-fe698e89c71d\" (UID: \"58c23db7-11cb-4ce2-a67e-fe698e89c71d\") " Oct 30 19:21:23.150409 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:23.149917 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84ca40c5-7033-48c4-bcec-e9ac7c034a4e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84ca40c5-7033-48c4-bcec-e9ac7c034a4e-model-cache\") pod \"84ca40c5-7033-48c4-bcec-e9ac7c034a4e\" (UID: \"84ca40c5-7033-48c4-bcec-e9ac7c034a4e\") " Oct 30 19:21:23.150409 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:23.149965 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58c23db7-11cb-4ce2-a67e-fe698e89c71d-tmp-dir" pod "58c23db7-11cb-4ce2-a67e-fe698e89c71d" (UID: "58c23db7-11cb-4ce2-a67e-fe698e89c71d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:21:23.150409 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:23.149976 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/943f7840-8d54-4845-b7ec-c0da72ba9c67-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/943f7840-8d54-4845-b7ec-c0da72ba9c67-model-cache\") pod \"943f7840-8d54-4845-b7ec-c0da72ba9c67\" (UID: \"943f7840-8d54-4845-b7ec-c0da72ba9c67\") " Oct 30 19:21:23.150813 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:23.150786 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/943f7840-8d54-4845-b7ec-c0da72ba9c67-model-cache" pod "943f7840-8d54-4845-b7ec-c0da72ba9c67" (UID: "943f7840-8d54-4845-b7ec-c0da72ba9c67"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:21:23.151011 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:23.150989 6960 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84ca40c5-7033-48c4-bcec-e9ac7c034a4e-home\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:21:23.151088 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:23.151016 6960 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84ca40c5-7033-48c4-bcec-e9ac7c034a4e-tmp-dir\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:21:23.151088 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:23.151032 6960 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58c23db7-11cb-4ce2-a67e-fe698e89c71d-tmp-dir\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:21:23.151088 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:23.151045 6960 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/943f7840-8d54-4845-b7ec-c0da72ba9c67-model-cache\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:21:23.151088 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:23.151060 6960 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58c23db7-11cb-4ce2-a67e-fe698e89c71d-model-cache\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:21:23.151389 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:23.151350 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/943f7840-8d54-4845-b7ec-c0da72ba9c67-tmp-dir" pod "943f7840-8d54-4845-b7ec-c0da72ba9c67" (UID: "943f7840-8d54-4845-b7ec-c0da72ba9c67"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:21:23.151893 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:23.151630 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/943f7840-8d54-4845-b7ec-c0da72ba9c67-home" pod "943f7840-8d54-4845-b7ec-c0da72ba9c67" (UID: "943f7840-8d54-4845-b7ec-c0da72ba9c67"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:21:23.152601 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:23.152231 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58c23db7-11cb-4ce2-a67e-fe698e89c71d-kube-api-access-hv4j5" pod "58c23db7-11cb-4ce2-a67e-fe698e89c71d" (UID: "58c23db7-11cb-4ce2-a67e-fe698e89c71d"). InnerVolumeSpecName "kube-api-access-hv4j5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:21:23.152601 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:23.152261 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58c23db7-11cb-4ce2-a67e-fe698e89c71d-dshm" pod "58c23db7-11cb-4ce2-a67e-fe698e89c71d" (UID: "58c23db7-11cb-4ce2-a67e-fe698e89c71d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:21:23.152601 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:23.152559 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84ca40c5-7033-48c4-bcec-e9ac7c034a4e-dshm" pod "84ca40c5-7033-48c4-bcec-e9ac7c034a4e" (UID: "84ca40c5-7033-48c4-bcec-e9ac7c034a4e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:21:23.152778 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:23.152757 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84ca40c5-7033-48c4-bcec-e9ac7c034a4e-model-cache" pod "84ca40c5-7033-48c4-bcec-e9ac7c034a4e" (UID: "84ca40c5-7033-48c4-bcec-e9ac7c034a4e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:21:23.152848 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:23.152832 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58c23db7-11cb-4ce2-a67e-fe698e89c71d-home" pod "58c23db7-11cb-4ce2-a67e-fe698e89c71d" (UID: "58c23db7-11cb-4ce2-a67e-fe698e89c71d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:21:23.153197 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:23.153137 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84ca40c5-7033-48c4-bcec-e9ac7c034a4e-kube-api-access-k5xlc" pod "84ca40c5-7033-48c4-bcec-e9ac7c034a4e" (UID: "84ca40c5-7033-48c4-bcec-e9ac7c034a4e"). InnerVolumeSpecName "kube-api-access-k5xlc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:21:23.154006 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:23.153983 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/943f7840-8d54-4845-b7ec-c0da72ba9c67-dshm" pod "943f7840-8d54-4845-b7ec-c0da72ba9c67" (UID: "943f7840-8d54-4845-b7ec-c0da72ba9c67"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:21:23.154611 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:23.154575 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84ca40c5-7033-48c4-bcec-e9ac7c034a4e-tls-certs" pod "84ca40c5-7033-48c4-bcec-e9ac7c034a4e" (UID: "84ca40c5-7033-48c4-bcec-e9ac7c034a4e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:21:23.154948 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:23.154918 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/943f7840-8d54-4845-b7ec-c0da72ba9c67-kube-api-access-x2xhz" pod "943f7840-8d54-4845-b7ec-c0da72ba9c67" (UID: "943f7840-8d54-4845-b7ec-c0da72ba9c67"). InnerVolumeSpecName "kube-api-access-x2xhz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:21:23.155816 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:23.155784 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/943f7840-8d54-4845-b7ec-c0da72ba9c67-tls-certs" pod "943f7840-8d54-4845-b7ec-c0da72ba9c67" (UID: "943f7840-8d54-4845-b7ec-c0da72ba9c67"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:21:23.156287 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:23.156235 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58c23db7-11cb-4ce2-a67e-fe698e89c71d-tls-certs" pod "58c23db7-11cb-4ce2-a67e-fe698e89c71d" (UID: "58c23db7-11cb-4ce2-a67e-fe698e89c71d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:21:23.252721 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:23.252619 6960 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84ca40c5-7033-48c4-bcec-e9ac7c034a4e-dshm\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:21:23.252721 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:23.252648 6960 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84ca40c5-7033-48c4-bcec-e9ac7c034a4e-tls-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:21:23.252721 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:23.252657 6960 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/943f7840-8d54-4845-b7ec-c0da72ba9c67-home\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:21:23.252721 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:23.252665 6960 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/943f7840-8d54-4845-b7ec-c0da72ba9c67-tmp-dir\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:21:23.252721 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:23.252673 6960 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/943f7840-8d54-4845-b7ec-c0da72ba9c67-dshm\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:21:23.252721 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:23.252681 6960 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58c23db7-11cb-4ce2-a67e-fe698e89c71d-home\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:21:23.252721 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:23.252690 6960 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58c23db7-11cb-4ce2-a67e-fe698e89c71d-tls-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:21:23.252721 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:23.252699 6960 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84ca40c5-7033-48c4-bcec-e9ac7c034a4e-model-cache\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:21:23.252721 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:23.252713 6960 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58c23db7-11cb-4ce2-a67e-fe698e89c71d-dshm\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:21:23.253418 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:23.252762 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hv4j5\" (UniqueName: \"kubernetes.io/projected/58c23db7-11cb-4ce2-a67e-fe698e89c71d-kube-api-access-hv4j5\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:21:23.253418 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:23.252783 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2xhz\" (UniqueName: \"kubernetes.io/projected/943f7840-8d54-4845-b7ec-c0da72ba9c67-kube-api-access-x2xhz\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:21:23.253418 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:23.252807 6960 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/943f7840-8d54-4845-b7ec-c0da72ba9c67-tls-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:21:23.253418 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:23.252820 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5xlc\" (UniqueName: \"kubernetes.io/projected/84ca40c5-7033-48c4-bcec-e9ac7c034a4e-kube-api-access-k5xlc\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:21:23.287582 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:23.287535 6960 generic.go:356] "Generic (PLEG): container finished" podID="943f7840-8d54-4845-b7ec-c0da72ba9c67" containerID="af2a3147a3a36da06b44805485145efe0622412f4c5e6c9f94c0b755a6811f2b" exitCode=0 Oct 30 19:21:23.287763 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:23.287596 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-aa6736/simulator-w1-aa6736-kserve-76985dcd5b-njvnn" Oct 30 19:21:23.287763 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:23.287605 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-aa6736/simulator-w1-aa6736-kserve-76985dcd5b-njvnn" event={"ID":"943f7840-8d54-4845-b7ec-c0da72ba9c67","Type":"ContainerDied","Data":"af2a3147a3a36da06b44805485145efe0622412f4c5e6c9f94c0b755a6811f2b"} Oct 30 19:21:23.287763 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:23.287668 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-aa6736/simulator-w1-aa6736-kserve-76985dcd5b-njvnn" event={"ID":"943f7840-8d54-4845-b7ec-c0da72ba9c67","Type":"ContainerDied","Data":"8f239e29fd9f496e358c7e13fee9ad23cf3b89c72830fe898756d3fb76d43055"} Oct 30 19:21:23.287763 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:23.287690 6960 scope.go:122] "RemoveContainer" containerID="af2a3147a3a36da06b44805485145efe0622412f4c5e6c9f94c0b755a6811f2b" Oct 30 19:21:23.292916 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:23.292882 6960 generic.go:356] "Generic (PLEG): container finished" podID="84ca40c5-7033-48c4-bcec-e9ac7c034a4e" containerID="3f5ce52a570bb1a7e8b84a47e71b27cbcb6454e8f194ce65a6e555ae7b5bb2b2" exitCode=0 Oct 30 19:21:23.293049 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:23.292999 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-aa6736/premium-w1-aa6736-kserve-6f986b66f7-nkfrg" event={"ID":"84ca40c5-7033-48c4-bcec-e9ac7c034a4e","Type":"ContainerDied","Data":"3f5ce52a570bb1a7e8b84a47e71b27cbcb6454e8f194ce65a6e555ae7b5bb2b2"} Oct 30 19:21:23.293049 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:23.293032 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-aa6736/premium-w1-aa6736-kserve-6f986b66f7-nkfrg" event={"ID":"84ca40c5-7033-48c4-bcec-e9ac7c034a4e","Type":"ContainerDied","Data":"8613f8c5399726db6bda0c14bc3635db951a6a22bddf75bc1f75d9855f9a6577"} Oct 30 19:21:23.293211 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:23.293100 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-aa6736/premium-w1-aa6736-kserve-6f986b66f7-nkfrg" Oct 30 19:21:23.296923 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:23.296887 6960 generic.go:356] "Generic (PLEG): container finished" podID="58c23db7-11cb-4ce2-a67e-fe698e89c71d" containerID="c5915f27d8d9f6e341312f052cc2e232ecd2170eeb1f9fc05a374f08aa593681" exitCode=0 Oct 30 19:21:23.297053 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:23.296921 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-aa6736/distinct-w1-aa6736-kserve-b86d7667b-smcbz" Oct 30 19:21:23.297053 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:23.296953 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-aa6736/distinct-w1-aa6736-kserve-b86d7667b-smcbz" event={"ID":"58c23db7-11cb-4ce2-a67e-fe698e89c71d","Type":"ContainerDied","Data":"c5915f27d8d9f6e341312f052cc2e232ecd2170eeb1f9fc05a374f08aa593681"} Oct 30 19:21:23.297053 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:23.296984 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-aa6736/distinct-w1-aa6736-kserve-b86d7667b-smcbz" event={"ID":"58c23db7-11cb-4ce2-a67e-fe698e89c71d","Type":"ContainerDied","Data":"c677e600b5c27479c68933e741ce1c4f82a50125af51deb399fdb712049103cd"} Oct 30 19:21:23.305339 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:23.305319 6960 scope.go:122] "RemoveContainer" containerID="af2a3147a3a36da06b44805485145efe0622412f4c5e6c9f94c0b755a6811f2b" Oct 30 19:21:23.305723 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:21:23.305695 6960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af2a3147a3a36da06b44805485145efe0622412f4c5e6c9f94c0b755a6811f2b\": container with ID starting with af2a3147a3a36da06b44805485145efe0622412f4c5e6c9f94c0b755a6811f2b not found: ID does not exist" containerID="af2a3147a3a36da06b44805485145efe0622412f4c5e6c9f94c0b755a6811f2b" Oct 30 19:21:23.305778 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:23.305739 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af2a3147a3a36da06b44805485145efe0622412f4c5e6c9f94c0b755a6811f2b"} err="failed to get container status \"af2a3147a3a36da06b44805485145efe0622412f4c5e6c9f94c0b755a6811f2b\": rpc error: code = NotFound desc = could not find container \"af2a3147a3a36da06b44805485145efe0622412f4c5e6c9f94c0b755a6811f2b\": container with ID starting with af2a3147a3a36da06b44805485145efe0622412f4c5e6c9f94c0b755a6811f2b not found: ID does not exist" Oct 30 19:21:23.305778 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:23.305764 6960 scope.go:122] "RemoveContainer" containerID="3f5ce52a570bb1a7e8b84a47e71b27cbcb6454e8f194ce65a6e555ae7b5bb2b2" Oct 30 19:21:23.315191 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:23.315166 6960 scope.go:122] "RemoveContainer" containerID="3f5ce52a570bb1a7e8b84a47e71b27cbcb6454e8f194ce65a6e555ae7b5bb2b2" Oct 30 19:21:23.315441 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:21:23.315423 6960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f5ce52a570bb1a7e8b84a47e71b27cbcb6454e8f194ce65a6e555ae7b5bb2b2\": container with ID starting with 3f5ce52a570bb1a7e8b84a47e71b27cbcb6454e8f194ce65a6e555ae7b5bb2b2 not found: ID does not exist" containerID="3f5ce52a570bb1a7e8b84a47e71b27cbcb6454e8f194ce65a6e555ae7b5bb2b2" Oct 30 19:21:23.315487 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:23.315450 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f5ce52a570bb1a7e8b84a47e71b27cbcb6454e8f194ce65a6e555ae7b5bb2b2"} err="failed to get container status \"3f5ce52a570bb1a7e8b84a47e71b27cbcb6454e8f194ce65a6e555ae7b5bb2b2\": rpc error: code = NotFound desc = could not find container \"3f5ce52a570bb1a7e8b84a47e71b27cbcb6454e8f194ce65a6e555ae7b5bb2b2\": container with ID starting with 3f5ce52a570bb1a7e8b84a47e71b27cbcb6454e8f194ce65a6e555ae7b5bb2b2 not found: ID does not exist" Oct 30 19:21:23.315487 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:23.315465 6960 scope.go:122] "RemoveContainer" containerID="c5915f27d8d9f6e341312f052cc2e232ecd2170eeb1f9fc05a374f08aa593681" Oct 30 19:21:23.321959 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:23.321934 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-aa6736/simulator-w1-aa6736-kserve-76985dcd5b-njvnn"] Oct 30 19:21:23.325660 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:23.325644 6960 scope.go:122] "RemoveContainer" containerID="c5915f27d8d9f6e341312f052cc2e232ecd2170eeb1f9fc05a374f08aa593681" Oct 30 19:21:23.325929 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:23.325908 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-aa6736/simulator-w1-aa6736-kserve-76985dcd5b-njvnn"] Oct 30 19:21:23.326024 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:21:23.325915 6960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5915f27d8d9f6e341312f052cc2e232ecd2170eeb1f9fc05a374f08aa593681\": container with ID starting with c5915f27d8d9f6e341312f052cc2e232ecd2170eeb1f9fc05a374f08aa593681 not found: ID does not exist" containerID="c5915f27d8d9f6e341312f052cc2e232ecd2170eeb1f9fc05a374f08aa593681" Oct 30 19:21:23.326024 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:23.325964 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5915f27d8d9f6e341312f052cc2e232ecd2170eeb1f9fc05a374f08aa593681"} err="failed to get container status \"c5915f27d8d9f6e341312f052cc2e232ecd2170eeb1f9fc05a374f08aa593681\": rpc error: code = NotFound desc = could not find container \"c5915f27d8d9f6e341312f052cc2e232ecd2170eeb1f9fc05a374f08aa593681\": container with ID starting with c5915f27d8d9f6e341312f052cc2e232ecd2170eeb1f9fc05a374f08aa593681 not found: ID does not exist" Oct 30 19:21:23.341133 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:23.341103 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-aa6736/premium-w1-aa6736-kserve-6f986b66f7-nkfrg"] Oct 30 19:21:23.345434 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:23.345415 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-aa6736/premium-w1-aa6736-kserve-6f986b66f7-nkfrg"] Oct 30 19:21:23.357268 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:23.357245 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-aa6736/distinct-w1-aa6736-kserve-b86d7667b-smcbz"] Oct 30 19:21:23.363719 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:23.363701 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-aa6736/distinct-w1-aa6736-kserve-b86d7667b-smcbz"] Oct 30 19:21:23.447389 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:23.447354 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:23.909787 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:23.909750 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-aa6736/distinct-w1-aa6736-kserve-b86d7667b-smcbz" podUID="58c23db7-11cb-4ce2-a67e-fe698e89c71d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 19:21:23.932658 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:21:23.932622 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 19:21:23.971478 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:23.971382 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-aa6736/simulator-w1-aa6736-kserve-76985dcd5b-njvnn" podUID="943f7840-8d54-4845-b7ec-c0da72ba9c67" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:21:24.505219 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:24.505181 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:24.593869 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:24.593837 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58c23db7-11cb-4ce2-a67e-fe698e89c71d" path="/var/lib/kubelet/pods/58c23db7-11cb-4ce2-a67e-fe698e89c71d/volumes" Oct 30 19:21:24.594429 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:24.594408 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84ca40c5-7033-48c4-bcec-e9ac7c034a4e" path="/var/lib/kubelet/pods/84ca40c5-7033-48c4-bcec-e9ac7c034a4e/volumes" Oct 30 19:21:24.594904 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:24.594886 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="943f7840-8d54-4845-b7ec-c0da72ba9c67" path="/var/lib/kubelet/pods/943f7840-8d54-4845-b7ec-c0da72ba9c67/volumes" Oct 30 19:21:25.491449 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:25.491415 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:26.451299 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:26.451259 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:27.575646 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:27.569092 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:28.021210 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:28.021119 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:28.725705 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:28.725667 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:28.784215 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:28.780302 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-03362d-655b6cd7f8-ndl7x" Oct 30 19:21:28.830186 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:28.830157 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-03362d-5df77d4775-tn9s2" Oct 30 19:21:28.928104 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:28.928068 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8690e67-62d5-4a1b-8632-a23a7d2f2843-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8690e67-62d5-4a1b-8632-a23a7d2f2843-plugins-config-volume\") pod \"c8690e67-62d5-4a1b-8632-a23a7d2f2843\" (UID: \"c8690e67-62d5-4a1b-8632-a23a7d2f2843\") " Oct 30 19:21:28.928343 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:28.928276 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8690e67-62d5-4a1b-8632-a23a7d2f2843-kube-api-access-qpc7q\" (UniqueName: \"kubernetes.io/projected/c8690e67-62d5-4a1b-8632-a23a7d2f2843-kube-api-access-qpc7q\") pod \"c8690e67-62d5-4a1b-8632-a23a7d2f2843\" (UID: \"c8690e67-62d5-4a1b-8632-a23a7d2f2843\") " Oct 30 19:21:28.928419 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:28.928390 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5a06a95-d251-46b6-889c-7e7d5e2ca925-kube-api-access-b9rwr\" (UniqueName: \"kubernetes.io/projected/b5a06a95-d251-46b6-889c-7e7d5e2ca925-kube-api-access-b9rwr\") pod \"b5a06a95-d251-46b6-889c-7e7d5e2ca925\" (UID: \"b5a06a95-d251-46b6-889c-7e7d5e2ca925\") " Oct 30 19:21:28.928480 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:28.928430 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8690e67-62d5-4a1b-8632-a23a7d2f2843-plugins-config-volume" pod "c8690e67-62d5-4a1b-8632-a23a7d2f2843" (UID: "c8690e67-62d5-4a1b-8632-a23a7d2f2843"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:21:28.928480 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:28.928463 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5a06a95-d251-46b6-889c-7e7d5e2ca925-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5a06a95-d251-46b6-889c-7e7d5e2ca925-plugins-config-volume\") pod \"b5a06a95-d251-46b6-889c-7e7d5e2ca925\" (UID: \"b5a06a95-d251-46b6-889c-7e7d5e2ca925\") " Oct 30 19:21:28.928865 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:28.928847 6960 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8690e67-62d5-4a1b-8632-a23a7d2f2843-plugins-config-volume\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:21:28.928938 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:28.928842 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5a06a95-d251-46b6-889c-7e7d5e2ca925-plugins-config-volume" pod "b5a06a95-d251-46b6-889c-7e7d5e2ca925" (UID: "b5a06a95-d251-46b6-889c-7e7d5e2ca925"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:21:28.930722 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:28.930699 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5a06a95-d251-46b6-889c-7e7d5e2ca925-kube-api-access-b9rwr" pod "b5a06a95-d251-46b6-889c-7e7d5e2ca925" (UID: "b5a06a95-d251-46b6-889c-7e7d5e2ca925"). InnerVolumeSpecName "kube-api-access-b9rwr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:21:28.930848 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:28.930769 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8690e67-62d5-4a1b-8632-a23a7d2f2843-kube-api-access-qpc7q" pod "c8690e67-62d5-4a1b-8632-a23a7d2f2843" (UID: "c8690e67-62d5-4a1b-8632-a23a7d2f2843"). InnerVolumeSpecName "kube-api-access-qpc7q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:21:28.979332 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:28.979247 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:29.030622 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:29.030256 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9rwr\" (UniqueName: \"kubernetes.io/projected/b5a06a95-d251-46b6-889c-7e7d5e2ca925-kube-api-access-b9rwr\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:21:29.030622 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:29.030284 6960 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5a06a95-d251-46b6-889c-7e7d5e2ca925-plugins-config-volume\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:21:29.030622 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:29.030294 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpc7q\" (UniqueName: \"kubernetes.io/projected/c8690e67-62d5-4a1b-8632-a23a7d2f2843-kube-api-access-qpc7q\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:21:29.134066 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:29.134017 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:29.341699 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:29.341666 6960 generic.go:356] "Generic (PLEG): container finished" podID="c8690e67-62d5-4a1b-8632-a23a7d2f2843" containerID="2de73ed0cef9082202870f240fb4c702b838bc4a07eb49a8f74f61a1eb528621" exitCode=0 Oct 30 19:21:29.341960 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:29.341934 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-03362d-5df77d4775-tn9s2" Oct 30 19:21:29.342983 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:29.342946 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-03362d-5df77d4775-tn9s2" event={"ID":"c8690e67-62d5-4a1b-8632-a23a7d2f2843","Type":"ContainerDied","Data":"2de73ed0cef9082202870f240fb4c702b838bc4a07eb49a8f74f61a1eb528621"} Oct 30 19:21:29.343095 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:29.343007 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-03362d-5df77d4775-tn9s2" event={"ID":"c8690e67-62d5-4a1b-8632-a23a7d2f2843","Type":"ContainerDied","Data":"da88029127bd3fad84607bb68f45bc54ec3152b32181d6723cc9207da7e5480d"} Oct 30 19:21:29.343095 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:29.343036 6960 scope.go:122] "RemoveContainer" containerID="2de73ed0cef9082202870f240fb4c702b838bc4a07eb49a8f74f61a1eb528621" Oct 30 19:21:29.344247 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:29.344225 6960 generic.go:356] "Generic (PLEG): container finished" podID="b5a06a95-d251-46b6-889c-7e7d5e2ca925" containerID="2470337503e5e11c7b374c3739b0d98a86e584f1f3db5fec12207fc2f6d2daca" exitCode=0 Oct 30 19:21:29.344299 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:29.344270 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-03362d-655b6cd7f8-ndl7x" event={"ID":"b5a06a95-d251-46b6-889c-7e7d5e2ca925","Type":"ContainerDied","Data":"2470337503e5e11c7b374c3739b0d98a86e584f1f3db5fec12207fc2f6d2daca"} Oct 30 19:21:29.344342 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:29.344309 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-03362d-655b6cd7f8-ndl7x" event={"ID":"b5a06a95-d251-46b6-889c-7e7d5e2ca925","Type":"ContainerDied","Data":"3659bd107e769d2b0cefd9e800cf2bab88f8d6db5dc5ac2a5d68d2465ebd4295"} Oct 30 19:21:29.344391 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:29.344378 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-03362d-655b6cd7f8-ndl7x" Oct 30 19:21:29.354923 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:29.354890 6960 scope.go:122] "RemoveContainer" containerID="2de73ed0cef9082202870f240fb4c702b838bc4a07eb49a8f74f61a1eb528621" Oct 30 19:21:29.355396 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:21:29.355363 6960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2de73ed0cef9082202870f240fb4c702b838bc4a07eb49a8f74f61a1eb528621\": container with ID starting with 2de73ed0cef9082202870f240fb4c702b838bc4a07eb49a8f74f61a1eb528621 not found: ID does not exist" containerID="2de73ed0cef9082202870f240fb4c702b838bc4a07eb49a8f74f61a1eb528621" Oct 30 19:21:29.355505 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:29.355407 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2de73ed0cef9082202870f240fb4c702b838bc4a07eb49a8f74f61a1eb528621"} err="failed to get container status \"2de73ed0cef9082202870f240fb4c702b838bc4a07eb49a8f74f61a1eb528621\": rpc error: code = NotFound desc = could not find container \"2de73ed0cef9082202870f240fb4c702b838bc4a07eb49a8f74f61a1eb528621\": container with ID starting with 2de73ed0cef9082202870f240fb4c702b838bc4a07eb49a8f74f61a1eb528621 not found: ID does not exist" Oct 30 19:21:29.355505 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:29.355430 6960 scope.go:122] "RemoveContainer" containerID="2470337503e5e11c7b374c3739b0d98a86e584f1f3db5fec12207fc2f6d2daca" Oct 30 19:21:29.366768 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:29.366747 6960 scope.go:122] "RemoveContainer" containerID="2470337503e5e11c7b374c3739b0d98a86e584f1f3db5fec12207fc2f6d2daca" Oct 30 19:21:29.367046 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:21:29.367030 6960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2470337503e5e11c7b374c3739b0d98a86e584f1f3db5fec12207fc2f6d2daca\": container with ID starting with 2470337503e5e11c7b374c3739b0d98a86e584f1f3db5fec12207fc2f6d2daca not found: ID does not exist" containerID="2470337503e5e11c7b374c3739b0d98a86e584f1f3db5fec12207fc2f6d2daca" Oct 30 19:21:29.367161 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:29.367054 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2470337503e5e11c7b374c3739b0d98a86e584f1f3db5fec12207fc2f6d2daca"} err="failed to get container status \"2470337503e5e11c7b374c3739b0d98a86e584f1f3db5fec12207fc2f6d2daca\": rpc error: code = NotFound desc = could not find container \"2470337503e5e11c7b374c3739b0d98a86e584f1f3db5fec12207fc2f6d2daca\": container with ID starting with 2470337503e5e11c7b374c3739b0d98a86e584f1f3db5fec12207fc2f6d2daca not found: ID does not exist" Oct 30 19:21:29.400522 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:29.400496 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-03362d-655b6cd7f8-ndl7x"] Oct 30 19:21:29.431489 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:29.431459 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-03362d-655b6cd7f8-ndl7x"] Oct 30 19:21:29.466716 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:29.466687 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-03362d-5df77d4775-tn9s2"] Oct 30 19:21:29.473686 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:29.473661 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-03362d-5df77d4775-tn9s2"] Oct 30 19:21:29.522228 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:29.522191 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:29.892169 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:29.887672 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:30.593981 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:30.593950 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5a06a95-d251-46b6-889c-7e7d5e2ca925" path="/var/lib/kubelet/pods/b5a06a95-d251-46b6-889c-7e7d5e2ca925/volumes" Oct 30 19:21:30.594375 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:30.594360 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8690e67-62d5-4a1b-8632-a23a7d2f2843" path="/var/lib/kubelet/pods/c8690e67-62d5-4a1b-8632-a23a7d2f2843/volumes" Oct 30 19:21:31.000429 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:31.000356 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:31.933982 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:31.933945 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:32.731880 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:32.731849 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:33.519735 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:33.519696 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:33.636028 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:33.635982 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:34.432923 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:34.432883 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:35.280499 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:35.280469 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:36.406773 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:36.406735 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:37.319256 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:37.319220 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:38.160070 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:38.159992 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:38.297043 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:38.297006 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:38.602585 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:38.602552 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:39.506397 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:39.505657 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-585dmj7jc"] Oct 30 19:21:39.581372 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:39.581328 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:39.708212 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:39.705027 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:40.056810 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:40.056772 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:40.260673 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:40.260598 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:40.401026 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:40.400933 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:40.531981 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:40.531932 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:40.637781 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:40.637697 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:40.786428 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:40.786319 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:40.898545 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:40.898512 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:41.314580 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:41.314541 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:41.860135 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:41.860100 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:42.784472 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:42.784443 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:43.625553 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:43.625517 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:44.506210 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:44.506166 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-585dmj7jc" podUID="21b44f0f-4b22-4369-9045-c9065515f816" containerName="payload-processing" containerID="cri-o://7bc156f4fa01301450cb1c9e4ce5e56eacf3c3a17297d79841e3c885b15b4be3" gracePeriod=25 Oct 30 19:21:44.740084 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:44.740040 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:44.789827 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:44.789735 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-585dmj7jc" Oct 30 19:21:44.853008 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:44.852968 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21b44f0f-4b22-4369-9045-c9065515f816-kube-api-access-2m46n\" (UniqueName: \"kubernetes.io/projected/21b44f0f-4b22-4369-9045-c9065515f816-kube-api-access-2m46n\") pod \"21b44f0f-4b22-4369-9045-c9065515f816\" (UID: \"21b44f0f-4b22-4369-9045-c9065515f816\") " Oct 30 19:21:44.853223 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:44.853047 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21b44f0f-4b22-4369-9045-c9065515f816-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/21b44f0f-4b22-4369-9045-c9065515f816-routing-overlay-volume\") pod \"21b44f0f-4b22-4369-9045-c9065515f816\" (UID: \"21b44f0f-4b22-4369-9045-c9065515f816\") " Oct 30 19:21:44.853223 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:44.853184 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21b44f0f-4b22-4369-9045-c9065515f816-provider-credentials\" (UniqueName: \"kubernetes.io/projected/21b44f0f-4b22-4369-9045-c9065515f816-provider-credentials\") pod \"21b44f0f-4b22-4369-9045-c9065515f816\" (UID: \"21b44f0f-4b22-4369-9045-c9065515f816\") " Oct 30 19:21:44.853471 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:44.853442 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21b44f0f-4b22-4369-9045-c9065515f816-routing-overlay-volume" pod "21b44f0f-4b22-4369-9045-c9065515f816" (UID: "21b44f0f-4b22-4369-9045-c9065515f816"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:21:44.853835 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:44.853462 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21b44f0f-4b22-4369-9045-c9065515f816-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21b44f0f-4b22-4369-9045-c9065515f816-plugins-config-volume\") pod \"21b44f0f-4b22-4369-9045-c9065515f816\" (UID: \"21b44f0f-4b22-4369-9045-c9065515f816\") " Oct 30 19:21:44.853835 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:44.853770 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21b44f0f-4b22-4369-9045-c9065515f816-plugins-config-volume" pod "21b44f0f-4b22-4369-9045-c9065515f816" (UID: "21b44f0f-4b22-4369-9045-c9065515f816"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:21:44.854570 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:44.854547 6960 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/21b44f0f-4b22-4369-9045-c9065515f816-routing-overlay-volume\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:21:44.854683 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:44.854576 6960 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21b44f0f-4b22-4369-9045-c9065515f816-plugins-config-volume\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:21:44.855810 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:44.855770 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21b44f0f-4b22-4369-9045-c9065515f816-kube-api-access-2m46n" pod "21b44f0f-4b22-4369-9045-c9065515f816" (UID: "21b44f0f-4b22-4369-9045-c9065515f816"). InnerVolumeSpecName "kube-api-access-2m46n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:21:44.855924 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:44.855830 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21b44f0f-4b22-4369-9045-c9065515f816-provider-credentials" pod "21b44f0f-4b22-4369-9045-c9065515f816" (UID: "21b44f0f-4b22-4369-9045-c9065515f816"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:21:44.956000 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:44.955957 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2m46n\" (UniqueName: \"kubernetes.io/projected/21b44f0f-4b22-4369-9045-c9065515f816-kube-api-access-2m46n\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:21:44.956000 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:44.955992 6960 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/21b44f0f-4b22-4369-9045-c9065515f816-provider-credentials\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:21:45.474960 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:45.474922 6960 generic.go:356] "Generic (PLEG): container finished" podID="21b44f0f-4b22-4369-9045-c9065515f816" containerID="7bc156f4fa01301450cb1c9e4ce5e56eacf3c3a17297d79841e3c885b15b4be3" exitCode=0 Oct 30 19:21:45.475186 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:45.475000 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-585dmj7jc" Oct 30 19:21:45.475186 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:45.475008 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-585dmj7jc" event={"ID":"21b44f0f-4b22-4369-9045-c9065515f816","Type":"ContainerDied","Data":"7bc156f4fa01301450cb1c9e4ce5e56eacf3c3a17297d79841e3c885b15b4be3"} Oct 30 19:21:45.475186 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:45.475077 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-585dmj7jc" event={"ID":"21b44f0f-4b22-4369-9045-c9065515f816","Type":"ContainerDied","Data":"89fe98e080941b9d933a99c93d129218fee1ada84fbc1db2c20494f672d43031"} Oct 30 19:21:45.475186 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:45.475105 6960 scope.go:122] "RemoveContainer" containerID="7bc156f4fa01301450cb1c9e4ce5e56eacf3c3a17297d79841e3c885b15b4be3" Oct 30 19:21:45.488468 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:45.488446 6960 scope.go:122] "RemoveContainer" containerID="7bc156f4fa01301450cb1c9e4ce5e56eacf3c3a17297d79841e3c885b15b4be3" Oct 30 19:21:45.488817 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:21:45.488796 6960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bc156f4fa01301450cb1c9e4ce5e56eacf3c3a17297d79841e3c885b15b4be3\": container with ID starting with 7bc156f4fa01301450cb1c9e4ce5e56eacf3c3a17297d79841e3c885b15b4be3 not found: ID does not exist" containerID="7bc156f4fa01301450cb1c9e4ce5e56eacf3c3a17297d79841e3c885b15b4be3" Oct 30 19:21:45.488955 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:45.488830 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bc156f4fa01301450cb1c9e4ce5e56eacf3c3a17297d79841e3c885b15b4be3"} err="failed to get container status \"7bc156f4fa01301450cb1c9e4ce5e56eacf3c3a17297d79841e3c885b15b4be3\": rpc error: code = NotFound desc = could not find container \"7bc156f4fa01301450cb1c9e4ce5e56eacf3c3a17297d79841e3c885b15b4be3\": container with ID starting with 7bc156f4fa01301450cb1c9e4ce5e56eacf3c3a17297d79841e3c885b15b4be3 not found: ID does not exist" Oct 30 19:21:45.505025 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:45.505001 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-585dmj7jc"] Oct 30 19:21:45.508426 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:45.508399 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-21d896/payload-processing-external-model-e2e-extmod-a-21d896-585dmj7jc"] Oct 30 19:21:45.779228 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:45.778930 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:46.593688 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:46.593654 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21b44f0f-4b22-4369-9045-c9065515f816" path="/var/lib/kubelet/pods/21b44f0f-4b22-4369-9045-c9065515f816/volumes" Oct 30 19:21:46.693445 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:46.693414 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:47.436960 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:47.436843 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:47.604269 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:47.604236 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:47.764022 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:47.763930 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:47.912236 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:47.912200 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:48.049242 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:48.047765 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:48.428288 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:48.428257 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:48.578956 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:48.578921 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:49.060624 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:49.060587 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:49.223963 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:49.223924 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:50.156547 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:50.156505 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:50.458752 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:50.458660 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:50.685625 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:50.685582 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:50.824857 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:50.824827 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-xg72j"] Oct 30 19:21:50.825059 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:50.825039 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-xg72j" podUID="4522747f-f5bf-481e-a35c-58e3475a3221" containerName="istio-proxy" containerID="cri-o://0e884b87ae4850a10a689779c11c2f61787d9364c30ed5bd1157ee399ce63096" gracePeriod=30 Oct 30 19:21:50.867539 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:50.867507 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:51.189696 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:51.189618 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-da70a5-5cb5894d96-ktj45"] Oct 30 19:21:51.190197 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:51.189866 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-da70a5-5cb5894d96-ktj45" podUID="5dc3436e-00d1-4ac1-9ea4-feaa1d92a15f" containerName="maas-api" containerID="cri-o://c810cb627ec671d1a208549ee2722ce552d182c28c53b2557d424b0b970cf0c8" gracePeriod=30 Oct 30 19:21:51.292678 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:51.292631 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:51.566660 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:51.565032 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:51.569990 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:51.569576 6960 generic.go:356] "Generic (PLEG): container finished" podID="5dc3436e-00d1-4ac1-9ea4-feaa1d92a15f" containerID="c810cb627ec671d1a208549ee2722ce552d182c28c53b2557d424b0b970cf0c8" exitCode=0 Oct 30 19:21:51.569990 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:51.569687 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-da70a5-5cb5894d96-ktj45" event={"ID":"5dc3436e-00d1-4ac1-9ea4-feaa1d92a15f","Type":"ContainerDied","Data":"c810cb627ec671d1a208549ee2722ce552d182c28c53b2557d424b0b970cf0c8"} Oct 30 19:21:51.569990 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:51.569734 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-da70a5-5cb5894d96-ktj45" event={"ID":"5dc3436e-00d1-4ac1-9ea4-feaa1d92a15f","Type":"ContainerDied","Data":"1205646afe42078afff642c3caa1dfa195382bcab55e0b4e90620d80d54f60ef"} Oct 30 19:21:51.569990 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:51.569759 6960 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1205646afe42078afff642c3caa1dfa195382bcab55e0b4e90620d80d54f60ef" Oct 30 19:21:51.577651 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:51.576921 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-da70a5-5cb5894d96-ktj45" Oct 30 19:21:51.693434 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:51.693403 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5dc3436e-00d1-4ac1-9ea4-feaa1d92a15f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5dc3436e-00d1-4ac1-9ea4-feaa1d92a15f-maas-api-tls\") pod \"5dc3436e-00d1-4ac1-9ea4-feaa1d92a15f\" (UID: \"5dc3436e-00d1-4ac1-9ea4-feaa1d92a15f\") " Oct 30 19:21:51.693434 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:51.693439 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5dc3436e-00d1-4ac1-9ea4-feaa1d92a15f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5dc3436e-00d1-4ac1-9ea4-feaa1d92a15f-metrics-certs\") pod \"5dc3436e-00d1-4ac1-9ea4-feaa1d92a15f\" (UID: \"5dc3436e-00d1-4ac1-9ea4-feaa1d92a15f\") " Oct 30 19:21:51.693646 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:51.693570 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dc3436e-00d1-4ac1-9ea4-feaa1d92a15f-kube-api-access-rtqmp\" (UniqueName: \"kubernetes.io/projected/5dc3436e-00d1-4ac1-9ea4-feaa1d92a15f-kube-api-access-rtqmp\") pod \"5dc3436e-00d1-4ac1-9ea4-feaa1d92a15f\" (UID: \"5dc3436e-00d1-4ac1-9ea4-feaa1d92a15f\") " Oct 30 19:21:51.693646 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:51.693643 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5dc3436e-00d1-4ac1-9ea4-feaa1d92a15f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dc3436e-00d1-4ac1-9ea4-feaa1d92a15f-openshift-service-ca\") pod \"5dc3436e-00d1-4ac1-9ea4-feaa1d92a15f\" (UID: \"5dc3436e-00d1-4ac1-9ea4-feaa1d92a15f\") " Oct 30 19:21:51.694106 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:51.694077 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dc3436e-00d1-4ac1-9ea4-feaa1d92a15f-openshift-service-ca" pod "5dc3436e-00d1-4ac1-9ea4-feaa1d92a15f" (UID: "5dc3436e-00d1-4ac1-9ea4-feaa1d92a15f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:21:51.694308 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:51.694292 6960 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dc3436e-00d1-4ac1-9ea4-feaa1d92a15f-openshift-service-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:21:51.695895 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:51.695869 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5dc3436e-00d1-4ac1-9ea4-feaa1d92a15f-maas-api-tls" pod "5dc3436e-00d1-4ac1-9ea4-feaa1d92a15f" (UID: "5dc3436e-00d1-4ac1-9ea4-feaa1d92a15f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:21:51.696458 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:51.696422 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dc3436e-00d1-4ac1-9ea4-feaa1d92a15f-kube-api-access-rtqmp" pod "5dc3436e-00d1-4ac1-9ea4-feaa1d92a15f" (UID: "5dc3436e-00d1-4ac1-9ea4-feaa1d92a15f"). InnerVolumeSpecName "kube-api-access-rtqmp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:21:51.696458 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:51.696441 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5dc3436e-00d1-4ac1-9ea4-feaa1d92a15f-metrics-certs" pod "5dc3436e-00d1-4ac1-9ea4-feaa1d92a15f" (UID: "5dc3436e-00d1-4ac1-9ea4-feaa1d92a15f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:21:51.733126 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:51.733080 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:51.796782 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:51.796692 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtqmp\" (UniqueName: \"kubernetes.io/projected/5dc3436e-00d1-4ac1-9ea4-feaa1d92a15f-kube-api-access-rtqmp\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:21:51.796782 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:51.796722 6960 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5dc3436e-00d1-4ac1-9ea4-feaa1d92a15f-maas-api-tls\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:21:51.796782 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:51.796735 6960 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5dc3436e-00d1-4ac1-9ea4-feaa1d92a15f-metrics-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:21:52.022120 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:52.022085 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:52.294957 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:52.294926 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:52.576114 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:52.576084 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-da70a5-5cb5894d96-ktj45" Oct 30 19:21:52.624503 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:52.624471 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-da70a5-5cb5894d96-ktj45"] Oct 30 19:21:52.632801 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:52.632768 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-da70a5-5cb5894d96-ktj45"] Oct 30 19:21:52.978822 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:52.978731 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:53.726160 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:53.726110 6960 scope.go:122] "RemoveContainer" containerID="c810cb627ec671d1a208549ee2722ce552d182c28c53b2557d424b0b970cf0c8" Oct 30 19:21:53.932431 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:21:53.932398 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 19:21:54.087047 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:54.087007 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:54.592379 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:54.592351 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5dc3436e-00d1-4ac1-9ea4-feaa1d92a15f" path="/var/lib/kubelet/pods/5dc3436e-00d1-4ac1-9ea4-feaa1d92a15f/volumes" Oct 30 19:21:54.721192 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:54.721136 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:55.064248 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:55.064213 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:55.885056 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:55.885027 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:56.102751 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:56.102696 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-xg72j" Oct 30 19:21:56.193539 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:56.193493 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:56.253754 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:56.253718 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4522747f-f5bf-481e-a35c-58e3475a3221-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4522747f-f5bf-481e-a35c-58e3475a3221-credential-socket\") pod \"4522747f-f5bf-481e-a35c-58e3475a3221\" (UID: \"4522747f-f5bf-481e-a35c-58e3475a3221\") " Oct 30 19:21:56.253969 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:56.253781 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4522747f-f5bf-481e-a35c-58e3475a3221-kube-api-access-9r8sh\" (UniqueName: \"kubernetes.io/projected/4522747f-f5bf-481e-a35c-58e3475a3221-kube-api-access-9r8sh\") pod \"4522747f-f5bf-481e-a35c-58e3475a3221\" (UID: \"4522747f-f5bf-481e-a35c-58e3475a3221\") " Oct 30 19:21:56.253969 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:56.253844 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4522747f-f5bf-481e-a35c-58e3475a3221-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4522747f-f5bf-481e-a35c-58e3475a3221-istiod-ca-cert\") pod \"4522747f-f5bf-481e-a35c-58e3475a3221\" (UID: \"4522747f-f5bf-481e-a35c-58e3475a3221\") " Oct 30 19:21:56.253969 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:56.253880 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4522747f-f5bf-481e-a35c-58e3475a3221-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4522747f-f5bf-481e-a35c-58e3475a3221-istio-podinfo\") pod \"4522747f-f5bf-481e-a35c-58e3475a3221\" (UID: \"4522747f-f5bf-481e-a35c-58e3475a3221\") " Oct 30 19:21:56.253969 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:56.253938 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4522747f-f5bf-481e-a35c-58e3475a3221-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4522747f-f5bf-481e-a35c-58e3475a3221-istio-envoy\") pod \"4522747f-f5bf-481e-a35c-58e3475a3221\" (UID: \"4522747f-f5bf-481e-a35c-58e3475a3221\") " Oct 30 19:21:56.255478 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:56.254015 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4522747f-f5bf-481e-a35c-58e3475a3221-istio-token\" (UniqueName: \"kubernetes.io/projected/4522747f-f5bf-481e-a35c-58e3475a3221-istio-token\") pod \"4522747f-f5bf-481e-a35c-58e3475a3221\" (UID: \"4522747f-f5bf-481e-a35c-58e3475a3221\") " Oct 30 19:21:56.255478 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:56.254080 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4522747f-f5bf-481e-a35c-58e3475a3221-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4522747f-f5bf-481e-a35c-58e3475a3221-workload-certs\") pod \"4522747f-f5bf-481e-a35c-58e3475a3221\" (UID: \"4522747f-f5bf-481e-a35c-58e3475a3221\") " Oct 30 19:21:56.255478 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:56.254118 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4522747f-f5bf-481e-a35c-58e3475a3221-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4522747f-f5bf-481e-a35c-58e3475a3221-istio-data\") pod \"4522747f-f5bf-481e-a35c-58e3475a3221\" (UID: \"4522747f-f5bf-481e-a35c-58e3475a3221\") " Oct 30 19:21:56.255478 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:56.254246 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4522747f-f5bf-481e-a35c-58e3475a3221-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4522747f-f5bf-481e-a35c-58e3475a3221-workload-socket\") pod \"4522747f-f5bf-481e-a35c-58e3475a3221\" (UID: \"4522747f-f5bf-481e-a35c-58e3475a3221\") " Oct 30 19:21:56.255478 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:56.255037 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4522747f-f5bf-481e-a35c-58e3475a3221-workload-socket" pod "4522747f-f5bf-481e-a35c-58e3475a3221" (UID: "4522747f-f5bf-481e-a35c-58e3475a3221"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:21:56.255478 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:56.255081 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4522747f-f5bf-481e-a35c-58e3475a3221-workload-certs" pod "4522747f-f5bf-481e-a35c-58e3475a3221" (UID: "4522747f-f5bf-481e-a35c-58e3475a3221"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:21:56.255478 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:56.255255 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4522747f-f5bf-481e-a35c-58e3475a3221-istio-data" pod "4522747f-f5bf-481e-a35c-58e3475a3221" (UID: "4522747f-f5bf-481e-a35c-58e3475a3221"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:21:56.255826 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:56.255620 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4522747f-f5bf-481e-a35c-58e3475a3221-istiod-ca-cert" pod "4522747f-f5bf-481e-a35c-58e3475a3221" (UID: "4522747f-f5bf-481e-a35c-58e3475a3221"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:21:56.256229 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:56.256121 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4522747f-f5bf-481e-a35c-58e3475a3221-credential-socket" pod "4522747f-f5bf-481e-a35c-58e3475a3221" (UID: "4522747f-f5bf-481e-a35c-58e3475a3221"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:21:56.260593 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:56.260567 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4522747f-f5bf-481e-a35c-58e3475a3221-kube-api-access-9r8sh" pod "4522747f-f5bf-481e-a35c-58e3475a3221" (UID: "4522747f-f5bf-481e-a35c-58e3475a3221"). InnerVolumeSpecName "kube-api-access-9r8sh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:21:56.263982 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:56.263957 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4522747f-f5bf-481e-a35c-58e3475a3221-istio-envoy" pod "4522747f-f5bf-481e-a35c-58e3475a3221" (UID: "4522747f-f5bf-481e-a35c-58e3475a3221"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:21:56.264194 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:56.264172 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4522747f-f5bf-481e-a35c-58e3475a3221-istio-token" pod "4522747f-f5bf-481e-a35c-58e3475a3221" (UID: "4522747f-f5bf-481e-a35c-58e3475a3221"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:21:56.265922 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:56.265902 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4522747f-f5bf-481e-a35c-58e3475a3221-istio-podinfo" pod "4522747f-f5bf-481e-a35c-58e3475a3221" (UID: "4522747f-f5bf-481e-a35c-58e3475a3221"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:21:56.333112 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:56.333061 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:56.355929 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:56.355866 6960 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4522747f-f5bf-481e-a35c-58e3475a3221-workload-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:21:56.355929 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:56.355893 6960 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4522747f-f5bf-481e-a35c-58e3475a3221-credential-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:21:56.355929 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:56.355908 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9r8sh\" (UniqueName: \"kubernetes.io/projected/4522747f-f5bf-481e-a35c-58e3475a3221-kube-api-access-9r8sh\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:21:56.355929 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:56.355924 6960 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4522747f-f5bf-481e-a35c-58e3475a3221-istiod-ca-cert\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:21:56.356155 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:56.355937 6960 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4522747f-f5bf-481e-a35c-58e3475a3221-istio-podinfo\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:21:56.356155 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:56.355951 6960 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4522747f-f5bf-481e-a35c-58e3475a3221-istio-envoy\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:21:56.356155 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:56.355962 6960 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4522747f-f5bf-481e-a35c-58e3475a3221-istio-token\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:21:56.356155 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:56.355976 6960 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4522747f-f5bf-481e-a35c-58e3475a3221-workload-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:21:56.356155 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:56.355988 6960 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4522747f-f5bf-481e-a35c-58e3475a3221-istio-data\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:21:56.443907 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:56.443865 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:56.603501 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:56.603470 6960 generic.go:356] "Generic (PLEG): container finished" podID="4522747f-f5bf-481e-a35c-58e3475a3221" containerID="0e884b87ae4850a10a689779c11c2f61787d9364c30ed5bd1157ee399ce63096" exitCode=0 Oct 30 19:21:56.603660 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:56.603524 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-xg72j" event={"ID":"4522747f-f5bf-481e-a35c-58e3475a3221","Type":"ContainerDied","Data":"0e884b87ae4850a10a689779c11c2f61787d9364c30ed5bd1157ee399ce63096"} Oct 30 19:21:56.603660 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:56.603534 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-xg72j" Oct 30 19:21:56.603660 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:56.603578 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-xg72j" event={"ID":"4522747f-f5bf-481e-a35c-58e3475a3221","Type":"ContainerDied","Data":"6d9f4a95e978dc48d54c8896f687d2be82479c246a4fc09669b37d6179747aa0"} Oct 30 19:21:56.603660 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:56.603609 6960 scope.go:122] "RemoveContainer" containerID="0e884b87ae4850a10a689779c11c2f61787d9364c30ed5bd1157ee399ce63096" Oct 30 19:21:56.614051 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:56.614035 6960 scope.go:122] "RemoveContainer" containerID="0e884b87ae4850a10a689779c11c2f61787d9364c30ed5bd1157ee399ce63096" Oct 30 19:21:56.614325 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:21:56.614309 6960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e884b87ae4850a10a689779c11c2f61787d9364c30ed5bd1157ee399ce63096\": container with ID starting with 0e884b87ae4850a10a689779c11c2f61787d9364c30ed5bd1157ee399ce63096 not found: ID does not exist" containerID="0e884b87ae4850a10a689779c11c2f61787d9364c30ed5bd1157ee399ce63096" Oct 30 19:21:56.614390 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:56.614331 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e884b87ae4850a10a689779c11c2f61787d9364c30ed5bd1157ee399ce63096"} err="failed to get container status \"0e884b87ae4850a10a689779c11c2f61787d9364c30ed5bd1157ee399ce63096\": rpc error: code = NotFound desc = could not find container \"0e884b87ae4850a10a689779c11c2f61787d9364c30ed5bd1157ee399ce63096\": container with ID starting with 0e884b87ae4850a10a689779c11c2f61787d9364c30ed5bd1157ee399ce63096 not found: ID does not exist" Oct 30 19:21:56.632417 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:56.632379 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-xg72j"] Oct 30 19:21:56.672074 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:56.672016 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-aa6736-openshift-default-6f74f55675-xg72j"] Oct 30 19:21:56.683394 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:56.683134 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:57.073696 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:57.073662 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:57.183553 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:57.183510 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:58.114980 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:58.114936 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:58.593949 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:58.593918 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4522747f-f5bf-481e-a35c-58e3475a3221" path="/var/lib/kubelet/pods/4522747f-f5bf-481e-a35c-58e3475a3221/volumes" Oct 30 19:21:58.870241 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:58.870085 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:58.987983 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:58.987941 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:59.114702 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:59.114672 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:59.528182 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:59.528114 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:59.771008 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:59.770963 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:21:59.936781 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:59.936737 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ng5vb"] Oct 30 19:21:59.936986 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:59.936956 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ng5vb" podUID="f1ef57ac-d33d-4998-b22b-87b2aed6ba8f" containerName="istio-proxy" containerID="cri-o://d9cf2cfc5db588847d9733b447ef0f784016c43123d8b0908aaf6c5d668993dd" gracePeriod=30 Oct 30 19:21:59.996673 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:21:59.996644 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:00.471487 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:00.471123 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:01.316294 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:01.316257 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:02.285265 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:02.285165 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:02.507189 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:02.507133 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:03.421195 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:03.421070 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:04.346269 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:04.346231 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:05.209346 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:05.209284 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ng5vb" Oct 30 19:22:05.284765 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:05.284736 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1ef57ac-d33d-4998-b22b-87b2aed6ba8f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1ef57ac-d33d-4998-b22b-87b2aed6ba8f-workload-certs\") pod \"f1ef57ac-d33d-4998-b22b-87b2aed6ba8f\" (UID: \"f1ef57ac-d33d-4998-b22b-87b2aed6ba8f\") " Oct 30 19:22:05.284952 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:05.284798 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f1ef57ac-d33d-4998-b22b-87b2aed6ba8f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1ef57ac-d33d-4998-b22b-87b2aed6ba8f-istio-podinfo\") pod \"f1ef57ac-d33d-4998-b22b-87b2aed6ba8f\" (UID: \"f1ef57ac-d33d-4998-b22b-87b2aed6ba8f\") " Oct 30 19:22:05.284952 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:05.284898 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1ef57ac-d33d-4998-b22b-87b2aed6ba8f-kube-api-access-zb686\" (UniqueName: \"kubernetes.io/projected/f1ef57ac-d33d-4998-b22b-87b2aed6ba8f-kube-api-access-zb686\") pod \"f1ef57ac-d33d-4998-b22b-87b2aed6ba8f\" (UID: \"f1ef57ac-d33d-4998-b22b-87b2aed6ba8f\") " Oct 30 19:22:05.285078 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:05.284978 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1ef57ac-d33d-4998-b22b-87b2aed6ba8f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1ef57ac-d33d-4998-b22b-87b2aed6ba8f-istio-envoy\") pod \"f1ef57ac-d33d-4998-b22b-87b2aed6ba8f\" (UID: \"f1ef57ac-d33d-4998-b22b-87b2aed6ba8f\") " Oct 30 19:22:05.285078 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:05.285010 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1ef57ac-d33d-4998-b22b-87b2aed6ba8f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1ef57ac-d33d-4998-b22b-87b2aed6ba8f-workload-socket\") pod \"f1ef57ac-d33d-4998-b22b-87b2aed6ba8f\" (UID: \"f1ef57ac-d33d-4998-b22b-87b2aed6ba8f\") " Oct 30 19:22:05.285078 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:05.285055 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1ef57ac-d33d-4998-b22b-87b2aed6ba8f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1ef57ac-d33d-4998-b22b-87b2aed6ba8f-istio-data\") pod \"f1ef57ac-d33d-4998-b22b-87b2aed6ba8f\" (UID: \"f1ef57ac-d33d-4998-b22b-87b2aed6ba8f\") " Oct 30 19:22:05.285262 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:05.285079 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1ef57ac-d33d-4998-b22b-87b2aed6ba8f-workload-certs" pod "f1ef57ac-d33d-4998-b22b-87b2aed6ba8f" (UID: "f1ef57ac-d33d-4998-b22b-87b2aed6ba8f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:22:05.285262 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:05.285113 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1ef57ac-d33d-4998-b22b-87b2aed6ba8f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1ef57ac-d33d-4998-b22b-87b2aed6ba8f-credential-socket\") pod \"f1ef57ac-d33d-4998-b22b-87b2aed6ba8f\" (UID: \"f1ef57ac-d33d-4998-b22b-87b2aed6ba8f\") " Oct 30 19:22:05.285262 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:05.285197 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1ef57ac-d33d-4998-b22b-87b2aed6ba8f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1ef57ac-d33d-4998-b22b-87b2aed6ba8f-istiod-ca-cert\") pod \"f1ef57ac-d33d-4998-b22b-87b2aed6ba8f\" (UID: \"f1ef57ac-d33d-4998-b22b-87b2aed6ba8f\") " Oct 30 19:22:05.285262 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:05.285241 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1ef57ac-d33d-4998-b22b-87b2aed6ba8f-istio-token\" (UniqueName: \"kubernetes.io/projected/f1ef57ac-d33d-4998-b22b-87b2aed6ba8f-istio-token\") pod \"f1ef57ac-d33d-4998-b22b-87b2aed6ba8f\" (UID: \"f1ef57ac-d33d-4998-b22b-87b2aed6ba8f\") " Oct 30 19:22:05.285471 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:05.285361 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1ef57ac-d33d-4998-b22b-87b2aed6ba8f-workload-socket" pod "f1ef57ac-d33d-4998-b22b-87b2aed6ba8f" (UID: "f1ef57ac-d33d-4998-b22b-87b2aed6ba8f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:22:05.285471 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:05.285355 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1ef57ac-d33d-4998-b22b-87b2aed6ba8f-istio-data" pod "f1ef57ac-d33d-4998-b22b-87b2aed6ba8f" (UID: "f1ef57ac-d33d-4998-b22b-87b2aed6ba8f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:22:05.285471 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:05.285447 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1ef57ac-d33d-4998-b22b-87b2aed6ba8f-credential-socket" pod "f1ef57ac-d33d-4998-b22b-87b2aed6ba8f" (UID: "f1ef57ac-d33d-4998-b22b-87b2aed6ba8f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:22:05.285775 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:05.285728 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1ef57ac-d33d-4998-b22b-87b2aed6ba8f-istiod-ca-cert" pod "f1ef57ac-d33d-4998-b22b-87b2aed6ba8f" (UID: "f1ef57ac-d33d-4998-b22b-87b2aed6ba8f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:22:05.285995 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:05.285856 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:05.286242 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:05.286219 6960 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1ef57ac-d33d-4998-b22b-87b2aed6ba8f-workload-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:05.286327 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:05.286251 6960 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1ef57ac-d33d-4998-b22b-87b2aed6ba8f-istio-data\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:05.286327 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:05.286267 6960 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1ef57ac-d33d-4998-b22b-87b2aed6ba8f-credential-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:05.286327 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:05.286282 6960 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1ef57ac-d33d-4998-b22b-87b2aed6ba8f-istiod-ca-cert\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:05.286327 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:05.286297 6960 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1ef57ac-d33d-4998-b22b-87b2aed6ba8f-workload-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:05.287681 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:05.287661 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1ef57ac-d33d-4998-b22b-87b2aed6ba8f-istio-envoy" pod "f1ef57ac-d33d-4998-b22b-87b2aed6ba8f" (UID: "f1ef57ac-d33d-4998-b22b-87b2aed6ba8f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:22:05.287681 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:05.287667 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1ef57ac-d33d-4998-b22b-87b2aed6ba8f-kube-api-access-zb686" pod "f1ef57ac-d33d-4998-b22b-87b2aed6ba8f" (UID: "f1ef57ac-d33d-4998-b22b-87b2aed6ba8f"). InnerVolumeSpecName "kube-api-access-zb686". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:22:05.288053 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:05.288033 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f1ef57ac-d33d-4998-b22b-87b2aed6ba8f-istio-podinfo" pod "f1ef57ac-d33d-4998-b22b-87b2aed6ba8f" (UID: "f1ef57ac-d33d-4998-b22b-87b2aed6ba8f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:22:05.288315 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:05.288294 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1ef57ac-d33d-4998-b22b-87b2aed6ba8f-istio-token" pod "f1ef57ac-d33d-4998-b22b-87b2aed6ba8f" (UID: "f1ef57ac-d33d-4998-b22b-87b2aed6ba8f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:22:05.387667 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:05.387635 6960 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1ef57ac-d33d-4998-b22b-87b2aed6ba8f-istio-envoy\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:05.387667 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:05.387665 6960 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1ef57ac-d33d-4998-b22b-87b2aed6ba8f-istio-token\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:05.387667 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:05.387675 6960 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1ef57ac-d33d-4998-b22b-87b2aed6ba8f-istio-podinfo\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:05.388116 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:05.387685 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zb686\" (UniqueName: \"kubernetes.io/projected/f1ef57ac-d33d-4998-b22b-87b2aed6ba8f-kube-api-access-zb686\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:05.666113 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:05.666081 6960 generic.go:356] "Generic (PLEG): container finished" podID="f1ef57ac-d33d-4998-b22b-87b2aed6ba8f" containerID="d9cf2cfc5db588847d9733b447ef0f784016c43123d8b0908aaf6c5d668993dd" exitCode=0 Oct 30 19:22:05.666339 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:05.666167 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ng5vb" Oct 30 19:22:05.666339 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:05.666188 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ng5vb" event={"ID":"f1ef57ac-d33d-4998-b22b-87b2aed6ba8f","Type":"ContainerDied","Data":"d9cf2cfc5db588847d9733b447ef0f784016c43123d8b0908aaf6c5d668993dd"} Oct 30 19:22:05.666339 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:05.666254 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ng5vb" event={"ID":"f1ef57ac-d33d-4998-b22b-87b2aed6ba8f","Type":"ContainerDied","Data":"a5920a594f4c8ede1e71734b6b45c73d366e23ef439f7e1a27398913f26a147c"} Oct 30 19:22:05.666339 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:05.666274 6960 scope.go:122] "RemoveContainer" containerID="d9cf2cfc5db588847d9733b447ef0f784016c43123d8b0908aaf6c5d668993dd" Oct 30 19:22:05.681345 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:05.681327 6960 scope.go:122] "RemoveContainer" containerID="d9cf2cfc5db588847d9733b447ef0f784016c43123d8b0908aaf6c5d668993dd" Oct 30 19:22:05.681673 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:22:05.681650 6960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9cf2cfc5db588847d9733b447ef0f784016c43123d8b0908aaf6c5d668993dd\": container with ID starting with d9cf2cfc5db588847d9733b447ef0f784016c43123d8b0908aaf6c5d668993dd not found: ID does not exist" containerID="d9cf2cfc5db588847d9733b447ef0f784016c43123d8b0908aaf6c5d668993dd" Oct 30 19:22:05.681793 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:05.681681 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9cf2cfc5db588847d9733b447ef0f784016c43123d8b0908aaf6c5d668993dd"} err="failed to get container status \"d9cf2cfc5db588847d9733b447ef0f784016c43123d8b0908aaf6c5d668993dd\": rpc error: code = NotFound desc = could not find container \"d9cf2cfc5db588847d9733b447ef0f784016c43123d8b0908aaf6c5d668993dd\": container with ID starting with d9cf2cfc5db588847d9733b447ef0f784016c43123d8b0908aaf6c5d668993dd not found: ID does not exist" Oct 30 19:22:05.693623 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:05.693584 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ng5vb"] Oct 30 19:22:05.697408 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:05.697384 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-da70a5-openshift-default-bb4f4b65-ng5vb"] Oct 30 19:22:06.165276 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:06.165240 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:06.283789 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:06.283752 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:06.395686 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:06.395651 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:06.522429 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:06.522352 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:06.593905 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:06.593868 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1ef57ac-d33d-4998-b22b-87b2aed6ba8f" path="/var/lib/kubelet/pods/f1ef57ac-d33d-4998-b22b-87b2aed6ba8f/volumes" Oct 30 19:22:06.880703 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:06.880667 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:07.298637 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:07.298548 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:07.577603 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:07.577570 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:07.698378 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:07.698349 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:08.229811 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:08.229777 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:09.203577 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:09.203544 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:10.044101 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:10.044060 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:10.284022 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:10.283986 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:10.439032 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:10.439004 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:10.580038 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:10.580005 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:10.984674 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:10.984635 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:11.653983 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:11.653948 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:12.609666 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:12.609628 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:13.665471 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:13.665440 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:14.283727 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:14.283691 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:14.387420 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:14.387385 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:15.332674 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:15.332635 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:16.136743 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:16.136707 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:17.103657 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:17.103618 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:18.007753 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:18.007717 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:18.780835 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:18.780797 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:18.953519 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:18.953483 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:19.134407 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:19.134377 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:19.418634 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:19.418531 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:20.185679 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:20.185646 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:21.243622 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:21.243584 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:22.627242 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:22.627205 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:23.517841 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:23.517806 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:23.931802 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:22:23.931771 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 19:22:24.442649 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:24.442606 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:25.266032 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:25.265999 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:26.147489 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:26.147451 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:26.979038 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:26.979002 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:27.928821 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:27.928784 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:28.967617 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:28.967581 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:29.099901 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:29.099817 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:29.423204 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:29.423108 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:29.596323 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:29.595882 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-extmod-shr-user-cce8b2/e2e-usage-sim-4bc3fa-6677768f7d-b5tnm"] Oct 30 19:22:29.596323 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:29.596125 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-extmod-shr-user-cce8b2/e2e-usage-sim-4bc3fa-6677768f7d-b5tnm" podUID="8e4832f5-ee56-4ea9-82c2-9fffed013c35" containerName="sim" containerID="cri-o://8fe7c236ca19c2856395cb6c8979b6a230699b2a90d827f8417d49f45c82ffa7" gracePeriod=30 Oct 30 19:22:29.690428 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:29.690342 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:29.804692 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:29.802330 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:29.826007 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:29.825971 6960 generic.go:356] "Generic (PLEG): container finished" podID="8e4832f5-ee56-4ea9-82c2-9fffed013c35" containerID="8fe7c236ca19c2856395cb6c8979b6a230699b2a90d827f8417d49f45c82ffa7" exitCode=0 Oct 30 19:22:29.826229 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:29.826024 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-cce8b2/e2e-usage-sim-4bc3fa-6677768f7d-b5tnm" event={"ID":"8e4832f5-ee56-4ea9-82c2-9fffed013c35","Type":"ContainerDied","Data":"8fe7c236ca19c2856395cb6c8979b6a230699b2a90d827f8417d49f45c82ffa7"} Oct 30 19:22:29.997536 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:29.997394 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:30.019959 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:30.019729 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-cce8b2/e2e-usage-sim-4bc3fa-6677768f7d-b5tnm" Oct 30 19:22:30.039981 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:30.039948 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e4832f5-ee56-4ea9-82c2-9fffed013c35-kube-api-access-hrx69\" (UniqueName: \"kubernetes.io/projected/8e4832f5-ee56-4ea9-82c2-9fffed013c35-kube-api-access-hrx69\") pod \"8e4832f5-ee56-4ea9-82c2-9fffed013c35\" (UID: \"8e4832f5-ee56-4ea9-82c2-9fffed013c35\") " Oct 30 19:22:30.041104 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:30.040176 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e4832f5-ee56-4ea9-82c2-9fffed013c35-tls\" (UniqueName: \"kubernetes.io/secret/8e4832f5-ee56-4ea9-82c2-9fffed013c35-tls\") pod \"8e4832f5-ee56-4ea9-82c2-9fffed013c35\" (UID: \"8e4832f5-ee56-4ea9-82c2-9fffed013c35\") " Oct 30 19:22:30.053412 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:30.053384 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e4832f5-ee56-4ea9-82c2-9fffed013c35-tls" pod "8e4832f5-ee56-4ea9-82c2-9fffed013c35" (UID: "8e4832f5-ee56-4ea9-82c2-9fffed013c35"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:22:30.055327 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:30.055299 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e4832f5-ee56-4ea9-82c2-9fffed013c35-kube-api-access-hrx69" pod "8e4832f5-ee56-4ea9-82c2-9fffed013c35" (UID: "8e4832f5-ee56-4ea9-82c2-9fffed013c35"). InnerVolumeSpecName "kube-api-access-hrx69". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:22:30.121836 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:30.121798 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:30.141473 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:30.141442 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrx69\" (UniqueName: \"kubernetes.io/projected/8e4832f5-ee56-4ea9-82c2-9fffed013c35-kube-api-access-hrx69\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:30.141473 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:30.141471 6960 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/8e4832f5-ee56-4ea9-82c2-9fffed013c35-tls\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:30.187055 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:30.187021 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-d72beb-665d6f48c5-gb2vl"] Oct 30 19:22:30.333849 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:30.333819 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:30.552614 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:30.552584 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:30.716550 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:30.716508 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:30.834206 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:30.834178 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-cce8b2/e2e-usage-sim-4bc3fa-6677768f7d-b5tnm" Oct 30 19:22:30.834206 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:30.834184 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-cce8b2/e2e-usage-sim-4bc3fa-6677768f7d-b5tnm" event={"ID":"8e4832f5-ee56-4ea9-82c2-9fffed013c35","Type":"ContainerDied","Data":"fcb2ad78970574c728c00c8b713ee5757118090dc95259e6fcc88d2da27770fc"} Oct 30 19:22:30.834422 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:30.834241 6960 scope.go:122] "RemoveContainer" containerID="8fe7c236ca19c2856395cb6c8979b6a230699b2a90d827f8417d49f45c82ffa7" Oct 30 19:22:30.853237 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:30.853157 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-extmod-shr-user-cce8b2/e2e-usage-sim-4bc3fa-6677768f7d-b5tnm"] Oct 30 19:22:30.855866 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:30.855838 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-extmod-shr-user-cce8b2/e2e-usage-sim-4bc3fa-6677768f7d-b5tnm"] Oct 30 19:22:31.135015 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:31.134921 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-d72beb-97d7945b8-lf4k8"] Oct 30 19:22:31.135507 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:31.135190 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d72beb-97d7945b8-lf4k8" podUID="ff4a8ee1-e818-4f1b-9b1e-f024237e3ec4" containerName="maas-api" containerID="cri-o://ff8db13b62e33c1494f560c71d3163b3765b81c54856ec89f0a27d0179b4a014" gracePeriod=30 Oct 30 19:22:31.208342 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:31.208299 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:31.481229 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:31.480876 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:31.586588 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:31.586559 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d72beb-97d7945b8-lf4k8" Oct 30 19:22:31.600188 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:31.599185 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:31.720153 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:31.720113 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff4a8ee1-e818-4f1b-9b1e-f024237e3ec4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff4a8ee1-e818-4f1b-9b1e-f024237e3ec4-openshift-service-ca\") pod \"ff4a8ee1-e818-4f1b-9b1e-f024237e3ec4\" (UID: \"ff4a8ee1-e818-4f1b-9b1e-f024237e3ec4\") " Oct 30 19:22:31.720400 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:31.720333 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff4a8ee1-e818-4f1b-9b1e-f024237e3ec4-kube-api-access-48nrp\" (UniqueName: \"kubernetes.io/projected/ff4a8ee1-e818-4f1b-9b1e-f024237e3ec4-kube-api-access-48nrp\") pod \"ff4a8ee1-e818-4f1b-9b1e-f024237e3ec4\" (UID: \"ff4a8ee1-e818-4f1b-9b1e-f024237e3ec4\") " Oct 30 19:22:31.720492 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:31.720412 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff4a8ee1-e818-4f1b-9b1e-f024237e3ec4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff4a8ee1-e818-4f1b-9b1e-f024237e3ec4-maas-api-tls\") pod \"ff4a8ee1-e818-4f1b-9b1e-f024237e3ec4\" (UID: \"ff4a8ee1-e818-4f1b-9b1e-f024237e3ec4\") " Oct 30 19:22:31.720564 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:31.720551 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff4a8ee1-e818-4f1b-9b1e-f024237e3ec4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff4a8ee1-e818-4f1b-9b1e-f024237e3ec4-metrics-certs\") pod \"ff4a8ee1-e818-4f1b-9b1e-f024237e3ec4\" (UID: \"ff4a8ee1-e818-4f1b-9b1e-f024237e3ec4\") " Oct 30 19:22:31.720633 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:31.720599 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff4a8ee1-e818-4f1b-9b1e-f024237e3ec4-openshift-service-ca" pod "ff4a8ee1-e818-4f1b-9b1e-f024237e3ec4" (UID: "ff4a8ee1-e818-4f1b-9b1e-f024237e3ec4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:22:31.721085 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:31.721058 6960 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff4a8ee1-e818-4f1b-9b1e-f024237e3ec4-openshift-service-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:31.723370 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:31.723339 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff4a8ee1-e818-4f1b-9b1e-f024237e3ec4-maas-api-tls" pod "ff4a8ee1-e818-4f1b-9b1e-f024237e3ec4" (UID: "ff4a8ee1-e818-4f1b-9b1e-f024237e3ec4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:22:31.723472 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:31.723427 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff4a8ee1-e818-4f1b-9b1e-f024237e3ec4-metrics-certs" pod "ff4a8ee1-e818-4f1b-9b1e-f024237e3ec4" (UID: "ff4a8ee1-e818-4f1b-9b1e-f024237e3ec4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:22:31.723796 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:31.723778 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff4a8ee1-e818-4f1b-9b1e-f024237e3ec4-kube-api-access-48nrp" pod "ff4a8ee1-e818-4f1b-9b1e-f024237e3ec4" (UID: "ff4a8ee1-e818-4f1b-9b1e-f024237e3ec4"). InnerVolumeSpecName "kube-api-access-48nrp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:22:31.753338 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:31.753102 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:31.822670 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:31.822637 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48nrp\" (UniqueName: \"kubernetes.io/projected/ff4a8ee1-e818-4f1b-9b1e-f024237e3ec4-kube-api-access-48nrp\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:31.822670 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:31.822665 6960 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff4a8ee1-e818-4f1b-9b1e-f024237e3ec4-maas-api-tls\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:31.822670 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:31.822675 6960 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff4a8ee1-e818-4f1b-9b1e-f024237e3ec4-metrics-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:31.841678 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:31.841647 6960 generic.go:356] "Generic (PLEG): container finished" podID="ff4a8ee1-e818-4f1b-9b1e-f024237e3ec4" containerID="ff8db13b62e33c1494f560c71d3163b3765b81c54856ec89f0a27d0179b4a014" exitCode=0 Oct 30 19:22:31.841847 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:31.841692 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d72beb-97d7945b8-lf4k8" Oct 30 19:22:31.841847 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:31.841732 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d72beb-97d7945b8-lf4k8" event={"ID":"ff4a8ee1-e818-4f1b-9b1e-f024237e3ec4","Type":"ContainerDied","Data":"ff8db13b62e33c1494f560c71d3163b3765b81c54856ec89f0a27d0179b4a014"} Oct 30 19:22:31.841847 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:31.841782 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d72beb-97d7945b8-lf4k8" event={"ID":"ff4a8ee1-e818-4f1b-9b1e-f024237e3ec4","Type":"ContainerDied","Data":"eb4cb155966038d38f9529341952fe698f3036db56974b5a177cbb039c7a6605"} Oct 30 19:22:31.841847 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:31.841812 6960 scope.go:122] "RemoveContainer" containerID="ff8db13b62e33c1494f560c71d3163b3765b81c54856ec89f0a27d0179b4a014" Oct 30 19:22:31.852358 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:31.852342 6960 scope.go:122] "RemoveContainer" containerID="ff8db13b62e33c1494f560c71d3163b3765b81c54856ec89f0a27d0179b4a014" Oct 30 19:22:31.852633 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:22:31.852617 6960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff8db13b62e33c1494f560c71d3163b3765b81c54856ec89f0a27d0179b4a014\": container with ID starting with ff8db13b62e33c1494f560c71d3163b3765b81c54856ec89f0a27d0179b4a014 not found: ID does not exist" containerID="ff8db13b62e33c1494f560c71d3163b3765b81c54856ec89f0a27d0179b4a014" Oct 30 19:22:31.852681 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:31.852642 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff8db13b62e33c1494f560c71d3163b3765b81c54856ec89f0a27d0179b4a014"} err="failed to get container status \"ff8db13b62e33c1494f560c71d3163b3765b81c54856ec89f0a27d0179b4a014\": rpc error: code = NotFound desc = could not find container \"ff8db13b62e33c1494f560c71d3163b3765b81c54856ec89f0a27d0179b4a014\": container with ID starting with ff8db13b62e33c1494f560c71d3163b3765b81c54856ec89f0a27d0179b4a014 not found: ID does not exist" Oct 30 19:22:31.872374 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:31.872336 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-d72beb-97d7945b8-lf4k8"] Oct 30 19:22:31.880866 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:31.880844 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-d72beb-97d7945b8-lf4k8"] Oct 30 19:22:32.148113 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:32.148070 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:32.606397 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:32.606362 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e4832f5-ee56-4ea9-82c2-9fffed013c35" path="/var/lib/kubelet/pods/8e4832f5-ee56-4ea9-82c2-9fffed013c35/volumes" Oct 30 19:22:32.606927 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:32.606906 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff4a8ee1-e818-4f1b-9b1e-f024237e3ec4" path="/var/lib/kubelet/pods/ff4a8ee1-e818-4f1b-9b1e-f024237e3ec4/volumes" Oct 30 19:22:32.640861 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:32.640818 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:33.576741 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:33.576708 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:34.415492 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:34.415457 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:35.187901 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:35.187860 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d72beb-665d6f48c5-gb2vl" podUID="1ff59ea2-88a1-4532-a214-6bafc573b44f" containerName="payload-pre-processing" containerID="cri-o://19c1dd549e8a86947ba2838f42518490a12ad96f41061fa6311ffc51f75ad64c" gracePeriod=25 Oct 30 19:22:35.329047 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:35.329009 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:36.424830 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:36.424802 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:37.419035 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:37.418998 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:38.225638 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:38.225602 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:38.389689 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:38.389657 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:38.536509 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:38.536370 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:38.708027 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:38.707984 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:39.217846 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:39.217799 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:39.360171 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:39.359912 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:39.535991 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:39.535885 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:39.673857 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:39.673811 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:39.815094 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:39.815061 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:39.940612 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:39.940576 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:40.070709 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:40.070627 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:40.284898 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:40.284858 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-4tbpd"] Oct 30 19:22:40.285100 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:40.285077 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-4tbpd" podUID="eb97b10a-d1c7-40a0-b182-2377facd98bc" containerName="istio-proxy" containerID="cri-o://fef5f4e4f15f14b3853fd2b5cf147978f49d35303e16d17f941adeb398a3e52f" gracePeriod=30 Oct 30 19:22:40.347753 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:40.347670 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:40.830271 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:40.830232 6960 patch_prober.go:29] interesting pod/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-4tbpd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:22:40.830781 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:40.830309 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-4tbpd" podUID="eb97b10a-d1c7-40a0-b182-2377facd98bc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:22:40.975178 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:40.975128 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:41.380085 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:41.380048 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:41.513870 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:41.513840 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:41.711046 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:41.710965 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-mw9h7"] Oct 30 19:22:41.711283 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:41.711234 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-mw9h7" podUID="afe56fb6-3076-47f1-92c7-a45b0426bdad" containerName="istio-proxy" containerID="cri-o://bfc0d3e2d12d5641b944a03b84362686bb3b0a7c50109ae5a6b82bf1ec158bc6" gracePeriod=30 Oct 30 19:22:41.776854 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:41.776818 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:41.975569 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:41.975458 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:43.174031 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:43.173988 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:43.365267 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:43.365233 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:44.237925 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:44.237881 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:44.436272 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:44.436231 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d72beb-665d6f48c5-gb2vl" podUID="1ff59ea2-88a1-4532-a214-6bafc573b44f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:22:44.715482 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:44.715441 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:44.887172 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:44.874571 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:45.315937 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:45.315906 6960 patch_prober.go:29] interesting pod/e2e-extmod-b-d72beb-openshift-default-869db9998f-mw9h7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:22:45.316365 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:45.315965 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-mw9h7" podUID="afe56fb6-3076-47f1-92c7-a45b0426bdad" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:22:45.559459 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:45.559433 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-4tbpd" Oct 30 19:22:45.633888 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:45.633828 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:45.719717 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:45.719689 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb97b10a-d1c7-40a0-b182-2377facd98bc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb97b10a-d1c7-40a0-b182-2377facd98bc-istiod-ca-cert\") pod \"eb97b10a-d1c7-40a0-b182-2377facd98bc\" (UID: \"eb97b10a-d1c7-40a0-b182-2377facd98bc\") " Oct 30 19:22:45.719893 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:45.719771 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb97b10a-d1c7-40a0-b182-2377facd98bc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb97b10a-d1c7-40a0-b182-2377facd98bc-istio-data\") pod \"eb97b10a-d1c7-40a0-b182-2377facd98bc\" (UID: \"eb97b10a-d1c7-40a0-b182-2377facd98bc\") " Oct 30 19:22:45.719893 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:45.719810 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb97b10a-d1c7-40a0-b182-2377facd98bc-istio-token\" (UniqueName: \"kubernetes.io/projected/eb97b10a-d1c7-40a0-b182-2377facd98bc-istio-token\") pod \"eb97b10a-d1c7-40a0-b182-2377facd98bc\" (UID: \"eb97b10a-d1c7-40a0-b182-2377facd98bc\") " Oct 30 19:22:45.719893 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:45.719887 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb97b10a-d1c7-40a0-b182-2377facd98bc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb97b10a-d1c7-40a0-b182-2377facd98bc-credential-socket\") pod \"eb97b10a-d1c7-40a0-b182-2377facd98bc\" (UID: \"eb97b10a-d1c7-40a0-b182-2377facd98bc\") " Oct 30 19:22:45.720853 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:45.719939 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb97b10a-d1c7-40a0-b182-2377facd98bc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb97b10a-d1c7-40a0-b182-2377facd98bc-workload-certs\") pod \"eb97b10a-d1c7-40a0-b182-2377facd98bc\" (UID: \"eb97b10a-d1c7-40a0-b182-2377facd98bc\") " Oct 30 19:22:45.720853 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:45.719971 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb97b10a-d1c7-40a0-b182-2377facd98bc-kube-api-access-s4gm7\" (UniqueName: \"kubernetes.io/projected/eb97b10a-d1c7-40a0-b182-2377facd98bc-kube-api-access-s4gm7\") pod \"eb97b10a-d1c7-40a0-b182-2377facd98bc\" (UID: \"eb97b10a-d1c7-40a0-b182-2377facd98bc\") " Oct 30 19:22:45.720853 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:45.720063 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb97b10a-d1c7-40a0-b182-2377facd98bc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb97b10a-d1c7-40a0-b182-2377facd98bc-istio-envoy\") pod \"eb97b10a-d1c7-40a0-b182-2377facd98bc\" (UID: \"eb97b10a-d1c7-40a0-b182-2377facd98bc\") " Oct 30 19:22:45.720853 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:45.720111 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb97b10a-d1c7-40a0-b182-2377facd98bc-istio-data" pod "eb97b10a-d1c7-40a0-b182-2377facd98bc" (UID: "eb97b10a-d1c7-40a0-b182-2377facd98bc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:22:45.720853 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:45.720130 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eb97b10a-d1c7-40a0-b182-2377facd98bc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb97b10a-d1c7-40a0-b182-2377facd98bc-istio-podinfo\") pod \"eb97b10a-d1c7-40a0-b182-2377facd98bc\" (UID: \"eb97b10a-d1c7-40a0-b182-2377facd98bc\") " Oct 30 19:22:45.720853 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:45.720123 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb97b10a-d1c7-40a0-b182-2377facd98bc-istiod-ca-cert" pod "eb97b10a-d1c7-40a0-b182-2377facd98bc" (UID: "eb97b10a-d1c7-40a0-b182-2377facd98bc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:22:45.720853 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:45.720202 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb97b10a-d1c7-40a0-b182-2377facd98bc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb97b10a-d1c7-40a0-b182-2377facd98bc-workload-socket\") pod \"eb97b10a-d1c7-40a0-b182-2377facd98bc\" (UID: \"eb97b10a-d1c7-40a0-b182-2377facd98bc\") " Oct 30 19:22:45.720853 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:45.720221 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb97b10a-d1c7-40a0-b182-2377facd98bc-credential-socket" pod "eb97b10a-d1c7-40a0-b182-2377facd98bc" (UID: "eb97b10a-d1c7-40a0-b182-2377facd98bc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:22:45.720853 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:45.720519 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb97b10a-d1c7-40a0-b182-2377facd98bc-workload-socket" pod "eb97b10a-d1c7-40a0-b182-2377facd98bc" (UID: "eb97b10a-d1c7-40a0-b182-2377facd98bc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:22:45.720853 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:45.720584 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb97b10a-d1c7-40a0-b182-2377facd98bc-workload-certs" pod "eb97b10a-d1c7-40a0-b182-2377facd98bc" (UID: "eb97b10a-d1c7-40a0-b182-2377facd98bc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:22:45.720853 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:45.720827 6960 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb97b10a-d1c7-40a0-b182-2377facd98bc-credential-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:45.720853 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:45.720848 6960 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb97b10a-d1c7-40a0-b182-2377facd98bc-workload-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:45.720853 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:45.720862 6960 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb97b10a-d1c7-40a0-b182-2377facd98bc-workload-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:45.720853 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:45.720878 6960 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb97b10a-d1c7-40a0-b182-2377facd98bc-istiod-ca-cert\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:45.721605 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:45.720891 6960 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb97b10a-d1c7-40a0-b182-2377facd98bc-istio-data\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:45.722410 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:45.722384 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb97b10a-d1c7-40a0-b182-2377facd98bc-istio-envoy" pod "eb97b10a-d1c7-40a0-b182-2377facd98bc" (UID: "eb97b10a-d1c7-40a0-b182-2377facd98bc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:22:45.722519 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:45.722426 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb97b10a-d1c7-40a0-b182-2377facd98bc-istio-token" pod "eb97b10a-d1c7-40a0-b182-2377facd98bc" (UID: "eb97b10a-d1c7-40a0-b182-2377facd98bc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:22:45.722890 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:45.722860 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eb97b10a-d1c7-40a0-b182-2377facd98bc-istio-podinfo" pod "eb97b10a-d1c7-40a0-b182-2377facd98bc" (UID: "eb97b10a-d1c7-40a0-b182-2377facd98bc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:22:45.722999 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:45.722899 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb97b10a-d1c7-40a0-b182-2377facd98bc-kube-api-access-s4gm7" pod "eb97b10a-d1c7-40a0-b182-2377facd98bc" (UID: "eb97b10a-d1c7-40a0-b182-2377facd98bc"). InnerVolumeSpecName "kube-api-access-s4gm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:22:45.822287 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:45.822209 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4gm7\" (UniqueName: \"kubernetes.io/projected/eb97b10a-d1c7-40a0-b182-2377facd98bc-kube-api-access-s4gm7\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:45.822287 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:45.822236 6960 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb97b10a-d1c7-40a0-b182-2377facd98bc-istio-envoy\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:45.822287 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:45.822246 6960 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb97b10a-d1c7-40a0-b182-2377facd98bc-istio-podinfo\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:45.822287 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:45.822255 6960 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb97b10a-d1c7-40a0-b182-2377facd98bc-istio-token\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:45.936311 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:45.936282 6960 generic.go:356] "Generic (PLEG): container finished" podID="eb97b10a-d1c7-40a0-b182-2377facd98bc" containerID="fef5f4e4f15f14b3853fd2b5cf147978f49d35303e16d17f941adeb398a3e52f" exitCode=0 Oct 30 19:22:45.936469 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:45.936317 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-4tbpd" event={"ID":"eb97b10a-d1c7-40a0-b182-2377facd98bc","Type":"ContainerDied","Data":"fef5f4e4f15f14b3853fd2b5cf147978f49d35303e16d17f941adeb398a3e52f"} Oct 30 19:22:45.936469 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:45.936335 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-4tbpd" Oct 30 19:22:45.936469 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:45.936356 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-4tbpd" event={"ID":"eb97b10a-d1c7-40a0-b182-2377facd98bc","Type":"ContainerDied","Data":"07c36d8f9893b9e15495ba10708c4009c3671ba0fc291b87dacac562d32dbb70"} Oct 30 19:22:45.936469 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:45.936377 6960 scope.go:122] "RemoveContainer" containerID="fef5f4e4f15f14b3853fd2b5cf147978f49d35303e16d17f941adeb398a3e52f" Oct 30 19:22:45.947611 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:45.947590 6960 scope.go:122] "RemoveContainer" containerID="fef5f4e4f15f14b3853fd2b5cf147978f49d35303e16d17f941adeb398a3e52f" Oct 30 19:22:45.947890 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:22:45.947869 6960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fef5f4e4f15f14b3853fd2b5cf147978f49d35303e16d17f941adeb398a3e52f\": container with ID starting with fef5f4e4f15f14b3853fd2b5cf147978f49d35303e16d17f941adeb398a3e52f not found: ID does not exist" containerID="fef5f4e4f15f14b3853fd2b5cf147978f49d35303e16d17f941adeb398a3e52f" Oct 30 19:22:45.947943 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:45.947898 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fef5f4e4f15f14b3853fd2b5cf147978f49d35303e16d17f941adeb398a3e52f"} err="failed to get container status \"fef5f4e4f15f14b3853fd2b5cf147978f49d35303e16d17f941adeb398a3e52f\": rpc error: code = NotFound desc = could not find container \"fef5f4e4f15f14b3853fd2b5cf147978f49d35303e16d17f941adeb398a3e52f\": container with ID starting with fef5f4e4f15f14b3853fd2b5cf147978f49d35303e16d17f941adeb398a3e52f not found: ID does not exist" Oct 30 19:22:45.962531 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:45.962510 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-4tbpd"] Oct 30 19:22:45.965473 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:45.965455 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-84c32a-openshift-default-8d49cb6bc-4tbpd"] Oct 30 19:22:46.565977 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:46.565944 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:46.604722 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:46.604678 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb97b10a-d1c7-40a0-b182-2377facd98bc" path="/var/lib/kubelet/pods/eb97b10a-d1c7-40a0-b182-2377facd98bc/volumes" Oct 30 19:22:46.945694 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:46.945661 6960 generic.go:356] "Generic (PLEG): container finished" podID="afe56fb6-3076-47f1-92c7-a45b0426bdad" containerID="bfc0d3e2d12d5641b944a03b84362686bb3b0a7c50109ae5a6b82bf1ec158bc6" exitCode=0 Oct 30 19:22:46.945878 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:46.945728 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-mw9h7" event={"ID":"afe56fb6-3076-47f1-92c7-a45b0426bdad","Type":"ContainerDied","Data":"bfc0d3e2d12d5641b944a03b84362686bb3b0a7c50109ae5a6b82bf1ec158bc6"} Oct 30 19:22:46.972952 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:46.972926 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-mw9h7" Oct 30 19:22:47.037348 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:47.037320 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afe56fb6-3076-47f1-92c7-a45b0426bdad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afe56fb6-3076-47f1-92c7-a45b0426bdad-workload-certs\") pod \"afe56fb6-3076-47f1-92c7-a45b0426bdad\" (UID: \"afe56fb6-3076-47f1-92c7-a45b0426bdad\") " Oct 30 19:22:47.037513 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:47.037362 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afe56fb6-3076-47f1-92c7-a45b0426bdad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afe56fb6-3076-47f1-92c7-a45b0426bdad-istio-envoy\") pod \"afe56fb6-3076-47f1-92c7-a45b0426bdad\" (UID: \"afe56fb6-3076-47f1-92c7-a45b0426bdad\") " Oct 30 19:22:47.037513 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:47.037404 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afe56fb6-3076-47f1-92c7-a45b0426bdad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afe56fb6-3076-47f1-92c7-a45b0426bdad-credential-socket\") pod \"afe56fb6-3076-47f1-92c7-a45b0426bdad\" (UID: \"afe56fb6-3076-47f1-92c7-a45b0426bdad\") " Oct 30 19:22:47.037513 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:47.037423 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afe56fb6-3076-47f1-92c7-a45b0426bdad-kube-api-access-7htz9\" (UniqueName: \"kubernetes.io/projected/afe56fb6-3076-47f1-92c7-a45b0426bdad-kube-api-access-7htz9\") pod \"afe56fb6-3076-47f1-92c7-a45b0426bdad\" (UID: \"afe56fb6-3076-47f1-92c7-a45b0426bdad\") " Oct 30 19:22:47.037513 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:47.037444 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/afe56fb6-3076-47f1-92c7-a45b0426bdad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afe56fb6-3076-47f1-92c7-a45b0426bdad-istio-podinfo\") pod \"afe56fb6-3076-47f1-92c7-a45b0426bdad\" (UID: \"afe56fb6-3076-47f1-92c7-a45b0426bdad\") " Oct 30 19:22:47.037513 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:47.037496 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afe56fb6-3076-47f1-92c7-a45b0426bdad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afe56fb6-3076-47f1-92c7-a45b0426bdad-istiod-ca-cert\") pod \"afe56fb6-3076-47f1-92c7-a45b0426bdad\" (UID: \"afe56fb6-3076-47f1-92c7-a45b0426bdad\") " Oct 30 19:22:47.037834 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:47.037555 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afe56fb6-3076-47f1-92c7-a45b0426bdad-istio-token\" (UniqueName: \"kubernetes.io/projected/afe56fb6-3076-47f1-92c7-a45b0426bdad-istio-token\") pod \"afe56fb6-3076-47f1-92c7-a45b0426bdad\" (UID: \"afe56fb6-3076-47f1-92c7-a45b0426bdad\") " Oct 30 19:22:47.037834 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:47.037585 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afe56fb6-3076-47f1-92c7-a45b0426bdad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afe56fb6-3076-47f1-92c7-a45b0426bdad-workload-socket\") pod \"afe56fb6-3076-47f1-92c7-a45b0426bdad\" (UID: \"afe56fb6-3076-47f1-92c7-a45b0426bdad\") " Oct 30 19:22:47.037834 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:47.037620 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afe56fb6-3076-47f1-92c7-a45b0426bdad-workload-certs" pod "afe56fb6-3076-47f1-92c7-a45b0426bdad" (UID: "afe56fb6-3076-47f1-92c7-a45b0426bdad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:22:47.037834 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:47.037738 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afe56fb6-3076-47f1-92c7-a45b0426bdad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afe56fb6-3076-47f1-92c7-a45b0426bdad-istio-data\") pod \"afe56fb6-3076-47f1-92c7-a45b0426bdad\" (UID: \"afe56fb6-3076-47f1-92c7-a45b0426bdad\") " Oct 30 19:22:47.038045 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:47.037976 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afe56fb6-3076-47f1-92c7-a45b0426bdad-workload-socket" pod "afe56fb6-3076-47f1-92c7-a45b0426bdad" (UID: "afe56fb6-3076-47f1-92c7-a45b0426bdad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:22:47.038358 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:47.038338 6960 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/afe56fb6-3076-47f1-92c7-a45b0426bdad-workload-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:47.038462 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:47.038362 6960 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/afe56fb6-3076-47f1-92c7-a45b0426bdad-workload-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:47.038508 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:47.038480 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afe56fb6-3076-47f1-92c7-a45b0426bdad-istiod-ca-cert" pod "afe56fb6-3076-47f1-92c7-a45b0426bdad" (UID: "afe56fb6-3076-47f1-92c7-a45b0426bdad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:22:47.038694 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:47.038664 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afe56fb6-3076-47f1-92c7-a45b0426bdad-istio-data" pod "afe56fb6-3076-47f1-92c7-a45b0426bdad" (UID: "afe56fb6-3076-47f1-92c7-a45b0426bdad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:22:47.038826 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:47.038805 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afe56fb6-3076-47f1-92c7-a45b0426bdad-credential-socket" pod "afe56fb6-3076-47f1-92c7-a45b0426bdad" (UID: "afe56fb6-3076-47f1-92c7-a45b0426bdad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:22:47.040846 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:47.040818 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afe56fb6-3076-47f1-92c7-a45b0426bdad-istio-token" pod "afe56fb6-3076-47f1-92c7-a45b0426bdad" (UID: "afe56fb6-3076-47f1-92c7-a45b0426bdad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:22:47.041195 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:47.041107 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/afe56fb6-3076-47f1-92c7-a45b0426bdad-istio-podinfo" pod "afe56fb6-3076-47f1-92c7-a45b0426bdad" (UID: "afe56fb6-3076-47f1-92c7-a45b0426bdad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:22:47.041372 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:47.041353 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afe56fb6-3076-47f1-92c7-a45b0426bdad-kube-api-access-7htz9" pod "afe56fb6-3076-47f1-92c7-a45b0426bdad" (UID: "afe56fb6-3076-47f1-92c7-a45b0426bdad"). InnerVolumeSpecName "kube-api-access-7htz9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:22:47.045471 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:47.045449 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afe56fb6-3076-47f1-92c7-a45b0426bdad-istio-envoy" pod "afe56fb6-3076-47f1-92c7-a45b0426bdad" (UID: "afe56fb6-3076-47f1-92c7-a45b0426bdad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:22:47.139398 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:47.139321 6960 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/afe56fb6-3076-47f1-92c7-a45b0426bdad-istiod-ca-cert\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:47.139398 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:47.139351 6960 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afe56fb6-3076-47f1-92c7-a45b0426bdad-istio-token\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:47.139398 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:47.139363 6960 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/afe56fb6-3076-47f1-92c7-a45b0426bdad-istio-data\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:47.139398 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:47.139376 6960 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/afe56fb6-3076-47f1-92c7-a45b0426bdad-istio-envoy\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:47.139398 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:47.139391 6960 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/afe56fb6-3076-47f1-92c7-a45b0426bdad-credential-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:47.139695 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:47.139404 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7htz9\" (UniqueName: \"kubernetes.io/projected/afe56fb6-3076-47f1-92c7-a45b0426bdad-kube-api-access-7htz9\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:47.139695 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:47.139417 6960 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/afe56fb6-3076-47f1-92c7-a45b0426bdad-istio-podinfo\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:47.554444 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:47.554349 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:47.955063 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:47.954990 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-mw9h7" Oct 30 19:22:47.955063 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:47.955026 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-mw9h7" event={"ID":"afe56fb6-3076-47f1-92c7-a45b0426bdad","Type":"ContainerDied","Data":"d9210f751ae600e9936eb62d0bfff862c735ee1c0aba660557636f5dc9f902a2"} Oct 30 19:22:47.955613 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:47.955093 6960 scope.go:122] "RemoveContainer" containerID="bfc0d3e2d12d5641b944a03b84362686bb3b0a7c50109ae5a6b82bf1ec158bc6" Oct 30 19:22:47.983070 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:47.983045 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-mw9h7"] Oct 30 19:22:47.985215 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:47.985193 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-d72beb-openshift-default-869db9998f-mw9h7"] Oct 30 19:22:48.486252 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:48.486219 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:48.593047 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:48.593013 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afe56fb6-3076-47f1-92c7-a45b0426bdad" path="/var/lib/kubelet/pods/afe56fb6-3076-47f1-92c7-a45b0426bdad/volumes" Oct 30 19:22:49.403952 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:49.403907 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:50.214850 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:50.214811 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:51.327584 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:51.327549 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:51.459436 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:51.459395 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:51.779926 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:51.778319 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:51.922351 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:51.922312 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:52.052523 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:52.052425 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:52.300886 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:52.300850 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:52.531185 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:52.531131 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:52.740999 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:52.740970 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:53.265377 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:53.265324 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:53.932206 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:22:53.932138 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 19:22:54.193314 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:54.193280 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:54.409450 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:54.409318 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:54.436348 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:54.436314 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d72beb-665d6f48c5-gb2vl" podUID="1ff59ea2-88a1-4532-a214-6bafc573b44f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:22:54.533372 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:54.533288 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:54.651119 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:54.647334 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:54.778881 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:54.778830 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:55.054723 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:55.054682 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:55.420378 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:55.420348 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:55.482960 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:55.482930 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d72beb-665d6f48c5-gb2vl" Oct 30 19:22:55.540429 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:55.540388 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ff59ea2-88a1-4532-a214-6bafc573b44f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ff59ea2-88a1-4532-a214-6bafc573b44f-plugins-config-volume\") pod \"1ff59ea2-88a1-4532-a214-6bafc573b44f\" (UID: \"1ff59ea2-88a1-4532-a214-6bafc573b44f\") " Oct 30 19:22:55.540623 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:55.540609 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ff59ea2-88a1-4532-a214-6bafc573b44f-kube-api-access-dgq7m\" (UniqueName: \"kubernetes.io/projected/1ff59ea2-88a1-4532-a214-6bafc573b44f-kube-api-access-dgq7m\") pod \"1ff59ea2-88a1-4532-a214-6bafc573b44f\" (UID: \"1ff59ea2-88a1-4532-a214-6bafc573b44f\") " Oct 30 19:22:55.540782 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:55.540755 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ff59ea2-88a1-4532-a214-6bafc573b44f-plugins-config-volume" pod "1ff59ea2-88a1-4532-a214-6bafc573b44f" (UID: "1ff59ea2-88a1-4532-a214-6bafc573b44f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:22:55.541025 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:55.541006 6960 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ff59ea2-88a1-4532-a214-6bafc573b44f-plugins-config-volume\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:55.542998 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:55.542962 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ff59ea2-88a1-4532-a214-6bafc573b44f-kube-api-access-dgq7m" pod "1ff59ea2-88a1-4532-a214-6bafc573b44f" (UID: "1ff59ea2-88a1-4532-a214-6bafc573b44f"). InnerVolumeSpecName "kube-api-access-dgq7m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:22:55.642603 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:55.642534 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgq7m\" (UniqueName: \"kubernetes.io/projected/1ff59ea2-88a1-4532-a214-6bafc573b44f-kube-api-access-dgq7m\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:22:56.006172 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:56.006083 6960 generic.go:356] "Generic (PLEG): container finished" podID="1ff59ea2-88a1-4532-a214-6bafc573b44f" containerID="19c1dd549e8a86947ba2838f42518490a12ad96f41061fa6311ffc51f75ad64c" exitCode=0 Oct 30 19:22:56.006172 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:56.006130 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d72beb-665d6f48c5-gb2vl" Oct 30 19:22:56.006355 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:56.006188 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d72beb-665d6f48c5-gb2vl" event={"ID":"1ff59ea2-88a1-4532-a214-6bafc573b44f","Type":"ContainerDied","Data":"19c1dd549e8a86947ba2838f42518490a12ad96f41061fa6311ffc51f75ad64c"} Oct 30 19:22:56.006355 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:56.006240 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d72beb-665d6f48c5-gb2vl" event={"ID":"1ff59ea2-88a1-4532-a214-6bafc573b44f","Type":"ContainerDied","Data":"264e21323ab8e261a4044bac58679c0268494a5ba420c116403e4242beace1a3"} Oct 30 19:22:56.006355 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:56.006263 6960 scope.go:122] "RemoveContainer" containerID="19c1dd549e8a86947ba2838f42518490a12ad96f41061fa6311ffc51f75ad64c" Oct 30 19:22:56.016014 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:56.015998 6960 scope.go:122] "RemoveContainer" containerID="19c1dd549e8a86947ba2838f42518490a12ad96f41061fa6311ffc51f75ad64c" Oct 30 19:22:56.016304 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:22:56.016282 6960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19c1dd549e8a86947ba2838f42518490a12ad96f41061fa6311ffc51f75ad64c\": container with ID starting with 19c1dd549e8a86947ba2838f42518490a12ad96f41061fa6311ffc51f75ad64c not found: ID does not exist" containerID="19c1dd549e8a86947ba2838f42518490a12ad96f41061fa6311ffc51f75ad64c" Oct 30 19:22:56.016376 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:56.016310 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19c1dd549e8a86947ba2838f42518490a12ad96f41061fa6311ffc51f75ad64c"} err="failed to get container status \"19c1dd549e8a86947ba2838f42518490a12ad96f41061fa6311ffc51f75ad64c\": rpc error: code = NotFound desc = could not find container \"19c1dd549e8a86947ba2838f42518490a12ad96f41061fa6311ffc51f75ad64c\": container with ID starting with 19c1dd549e8a86947ba2838f42518490a12ad96f41061fa6311ffc51f75ad64c not found: ID does not exist" Oct 30 19:22:56.049281 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:56.049258 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-d72beb-665d6f48c5-gb2vl"] Oct 30 19:22:56.056201 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:56.056181 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-d72beb-665d6f48c5-gb2vl"] Oct 30 19:22:56.496623 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:56.496567 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:56.593684 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:56.593652 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ff59ea2-88a1-4532-a214-6bafc573b44f" path="/var/lib/kubelet/pods/1ff59ea2-88a1-4532-a214-6bafc573b44f/volumes" Oct 30 19:22:57.417041 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:57.417009 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:58.262327 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:58.262292 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:22:59.206957 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:22:59.206920 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:00.376475 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:00.376443 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:01.197718 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:01.197686 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:02.027752 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:02.027720 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:02.194322 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:02.194290 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:02.315597 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:02.315567 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:02.644725 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:02.644639 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:02.969361 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:02.969258 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:03.298732 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:03.298641 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:03.778304 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:03.778267 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:03.939707 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:03.939668 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:04.346182 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:04.346123 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:04.677903 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:04.677809 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:05.206095 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:05.206054 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:06.130934 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:06.130898 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:07.174863 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:07.174820 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:08.213418 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:08.213387 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:08.373582 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:08.373547 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:09.341183 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:09.341131 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:09.511725 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:09.511693 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:09.655217 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:09.655099 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:10.335336 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:10.335301 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:11.220249 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:11.220209 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:12.129817 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:12.129781 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:12.917132 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:12.917088 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:13.464239 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:13.464203 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:13.733492 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:13.733405 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:14.124853 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:14.124820 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:14.916683 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:14.916641 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:15.730710 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:15.730678 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:15.916715 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:15.916677 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:16.558725 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:16.558685 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:16.775123 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:16.775037 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:16.909498 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:16.909414 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:17.085881 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:17.085852 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:17.535049 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:17.535009 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:17.968002 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:17.967967 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:18.822240 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:18.822207 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:19.713183 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:19.713124 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:20.609657 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:20.609627 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:21.611747 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:21.611712 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:22.402056 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:22.402022 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:23.397131 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:23.397091 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:23.932869 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:23:23.932835 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 19:23:24.257571 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:24.257476 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:25.452041 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:25.452003 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:25.786916 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:25.786818 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:25.919425 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:25.919393 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:26.219346 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:26.219300 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:26.515829 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:26.515741 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:26.716594 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:26.716551 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:27.515955 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:27.515925 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:28.255074 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:28.255043 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:28.379996 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:28.379954 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:29.336009 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:29.335975 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:30.129179 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:30.129100 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:30.286784 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:30.284125 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:31.101607 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:31.101577 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:31.647255 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:31.647223 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-7a8fa2/distinct-2-w3-7a8fa2-kserve-677bb44d8d-fg55k"] Oct 30 19:23:31.666123 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:31.666011 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-7a8fa2/premium-w3-7a8fa2-kserve-6b6c5f6fb8-n5wtq"] Oct 30 19:23:31.675969 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:31.675943 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-7a8fa2/simulator-w3-7a8fa2-kserve-78587cb8fc-6pvsz"] Oct 30 19:23:31.777781 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:31.777747 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:31.901929 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:31.901829 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:32.173818 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:32.173732 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:33.010505 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:33.010464 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:34.017066 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:34.016980 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:35.016342 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:35.016296 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:35.808213 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:35.808168 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:36.510604 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:36.510553 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:36.634947 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:36.634899 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:36.963589 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:36.963556 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-7a8fa2-778fb96bb-tlvqh"] Oct 30 19:23:37.043710 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:37.043681 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:37.160657 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:37.160612 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:37.522641 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:37.522606 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:38.350598 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:38.350540 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:39.411719 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:39.411691 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:40.392633 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:40.392594 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:41.223907 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:41.223872 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:41.963919 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:41.963862 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-7a8fa2-778fb96bb-tlvqh" podUID="67a3f38a-10e8-4826-925c-520ce8297f11" containerName="payload-processing" containerID="cri-o://e5d7327234db62163358bdb4e01d1231e9e3d2ea60891475b591eed63452145f" gracePeriod=25 Oct 30 19:23:42.193503 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:42.193461 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:43.008387 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:43.008347 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:43.802594 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:43.802559 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:44.908289 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:44.908250 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:45.787726 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:45.787691 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:46.669672 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:46.669608 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-2-w3-7a8fa2-kserve-677bb44d8d-fg55k" podUID="e8701242-b59c-43b1-9c0a-011635edd419" containerName="main" containerID="cri-o://da8c7e67cc7d7dbc66bb234713ee5a5e9a629cf12bd94737827fc249f1996cce" gracePeriod=45 Oct 30 19:23:46.693568 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:46.693533 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-7a8fa2/premium-w3-7a8fa2-kserve-6b6c5f6fb8-n5wtq" podUID="f51fd746-cd85-4608-96a6-f54afa0ec34d" containerName="main" containerID="cri-o://7272ada118e39fc2b5c37624c620ca1ffdf53ff85c9374e52e94e678b28b5c98" gracePeriod=45 Oct 30 19:23:46.700490 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:46.700462 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-7a8fa2/simulator-w3-7a8fa2-kserve-78587cb8fc-6pvsz" podUID="cc64ebb9-44e9-435a-9db0-d53ce5f139ff" containerName="main" containerID="cri-o://1971b54e599f3ec447c9fc92d5fa8bde1edea9aebb77a9805e45805e6765d9d7" gracePeriod=45 Oct 30 19:23:46.851579 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:46.851544 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:46.995503 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:46.995471 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-2-w3-7a8fa2-kserve-677bb44d8d-fg55k" Oct 30 19:23:47.009460 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:47.003067 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-7a8fa2/premium-w3-7a8fa2-kserve-6b6c5f6fb8-n5wtq" Oct 30 19:23:47.025515 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:47.025488 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-7a8fa2/simulator-w3-7a8fa2-kserve-78587cb8fc-6pvsz" Oct 30 19:23:47.150170 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:47.150090 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8701242-b59c-43b1-9c0a-011635edd419-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8701242-b59c-43b1-9c0a-011635edd419-model-cache\") pod \"e8701242-b59c-43b1-9c0a-011635edd419\" (UID: \"e8701242-b59c-43b1-9c0a-011635edd419\") " Oct 30 19:23:47.150456 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:47.150189 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f51fd746-cd85-4608-96a6-f54afa0ec34d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f51fd746-cd85-4608-96a6-f54afa0ec34d-dshm\") pod \"f51fd746-cd85-4608-96a6-f54afa0ec34d\" (UID: \"f51fd746-cd85-4608-96a6-f54afa0ec34d\") " Oct 30 19:23:47.150456 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:47.150392 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8701242-b59c-43b1-9c0a-011635edd419-model-cache" pod "e8701242-b59c-43b1-9c0a-011635edd419" (UID: "e8701242-b59c-43b1-9c0a-011635edd419"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:23:47.150615 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:47.150485 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc64ebb9-44e9-435a-9db0-d53ce5f139ff-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc64ebb9-44e9-435a-9db0-d53ce5f139ff-tmp-dir\") pod \"cc64ebb9-44e9-435a-9db0-d53ce5f139ff\" (UID: \"cc64ebb9-44e9-435a-9db0-d53ce5f139ff\") " Oct 30 19:23:47.150615 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:47.150515 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f51fd746-cd85-4608-96a6-f54afa0ec34d-tls-certs\" (UniqueName: \"kubernetes.io/secret/f51fd746-cd85-4608-96a6-f54afa0ec34d-tls-certs\") pod \"f51fd746-cd85-4608-96a6-f54afa0ec34d\" (UID: \"f51fd746-cd85-4608-96a6-f54afa0ec34d\") " Oct 30 19:23:47.150615 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:47.150538 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc64ebb9-44e9-435a-9db0-d53ce5f139ff-kube-api-access-lg4ll\" (UniqueName: \"kubernetes.io/projected/cc64ebb9-44e9-435a-9db0-d53ce5f139ff-kube-api-access-lg4ll\") pod \"cc64ebb9-44e9-435a-9db0-d53ce5f139ff\" (UID: \"cc64ebb9-44e9-435a-9db0-d53ce5f139ff\") " Oct 30 19:23:47.150615 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:47.150566 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f51fd746-cd85-4608-96a6-f54afa0ec34d-kube-api-access-kl582\" (UniqueName: \"kubernetes.io/projected/f51fd746-cd85-4608-96a6-f54afa0ec34d-kube-api-access-kl582\") pod \"f51fd746-cd85-4608-96a6-f54afa0ec34d\" (UID: \"f51fd746-cd85-4608-96a6-f54afa0ec34d\") " Oct 30 19:23:47.150831 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:47.150641 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f51fd746-cd85-4608-96a6-f54afa0ec34d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f51fd746-cd85-4608-96a6-f54afa0ec34d-tmp-dir\") pod \"f51fd746-cd85-4608-96a6-f54afa0ec34d\" (UID: \"f51fd746-cd85-4608-96a6-f54afa0ec34d\") " Oct 30 19:23:47.150831 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:47.150707 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8701242-b59c-43b1-9c0a-011635edd419-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8701242-b59c-43b1-9c0a-011635edd419-tmp-dir\") pod \"e8701242-b59c-43b1-9c0a-011635edd419\" (UID: \"e8701242-b59c-43b1-9c0a-011635edd419\") " Oct 30 19:23:47.150831 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:47.150764 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8701242-b59c-43b1-9c0a-011635edd419-kube-api-access-r822d\" (UniqueName: \"kubernetes.io/projected/e8701242-b59c-43b1-9c0a-011635edd419-kube-api-access-r822d\") pod \"e8701242-b59c-43b1-9c0a-011635edd419\" (UID: \"e8701242-b59c-43b1-9c0a-011635edd419\") " Oct 30 19:23:47.150831 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:47.150795 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc64ebb9-44e9-435a-9db0-d53ce5f139ff-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc64ebb9-44e9-435a-9db0-d53ce5f139ff-model-cache\") pod \"cc64ebb9-44e9-435a-9db0-d53ce5f139ff\" (UID: \"cc64ebb9-44e9-435a-9db0-d53ce5f139ff\") " Oct 30 19:23:47.150831 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:47.150796 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc64ebb9-44e9-435a-9db0-d53ce5f139ff-tmp-dir" pod "cc64ebb9-44e9-435a-9db0-d53ce5f139ff" (UID: "cc64ebb9-44e9-435a-9db0-d53ce5f139ff"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:23:47.151192 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:47.150836 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f51fd746-cd85-4608-96a6-f54afa0ec34d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f51fd746-cd85-4608-96a6-f54afa0ec34d-model-cache\") pod \"f51fd746-cd85-4608-96a6-f54afa0ec34d\" (UID: \"f51fd746-cd85-4608-96a6-f54afa0ec34d\") " Oct 30 19:23:47.151192 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:47.150897 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8701242-b59c-43b1-9c0a-011635edd419-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8701242-b59c-43b1-9c0a-011635edd419-dshm\") pod \"e8701242-b59c-43b1-9c0a-011635edd419\" (UID: \"e8701242-b59c-43b1-9c0a-011635edd419\") " Oct 30 19:23:47.151192 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:47.150940 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8701242-b59c-43b1-9c0a-011635edd419-home\" (UniqueName: \"kubernetes.io/empty-dir/e8701242-b59c-43b1-9c0a-011635edd419-home\") pod \"e8701242-b59c-43b1-9c0a-011635edd419\" (UID: \"e8701242-b59c-43b1-9c0a-011635edd419\") " Oct 30 19:23:47.151192 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:47.150982 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8701242-b59c-43b1-9c0a-011635edd419-tls-certs\" (UniqueName: \"kubernetes.io/secret/e8701242-b59c-43b1-9c0a-011635edd419-tls-certs\") pod \"e8701242-b59c-43b1-9c0a-011635edd419\" (UID: \"e8701242-b59c-43b1-9c0a-011635edd419\") " Oct 30 19:23:47.151192 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:47.151016 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8701242-b59c-43b1-9c0a-011635edd419-tmp-dir" pod "e8701242-b59c-43b1-9c0a-011635edd419" (UID: "e8701242-b59c-43b1-9c0a-011635edd419"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:23:47.151192 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:47.151018 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc64ebb9-44e9-435a-9db0-d53ce5f139ff-home\" (UniqueName: \"kubernetes.io/empty-dir/cc64ebb9-44e9-435a-9db0-d53ce5f139ff-home\") pod \"cc64ebb9-44e9-435a-9db0-d53ce5f139ff\" (UID: \"cc64ebb9-44e9-435a-9db0-d53ce5f139ff\") " Oct 30 19:23:47.151192 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:47.151119 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc64ebb9-44e9-435a-9db0-d53ce5f139ff-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc64ebb9-44e9-435a-9db0-d53ce5f139ff-dshm\") pod \"cc64ebb9-44e9-435a-9db0-d53ce5f139ff\" (UID: \"cc64ebb9-44e9-435a-9db0-d53ce5f139ff\") " Oct 30 19:23:47.151192 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:47.151185 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f51fd746-cd85-4608-96a6-f54afa0ec34d-home\" (UniqueName: \"kubernetes.io/empty-dir/f51fd746-cd85-4608-96a6-f54afa0ec34d-home\") pod \"f51fd746-cd85-4608-96a6-f54afa0ec34d\" (UID: \"f51fd746-cd85-4608-96a6-f54afa0ec34d\") " Oct 30 19:23:47.151821 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:47.151222 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc64ebb9-44e9-435a-9db0-d53ce5f139ff-tls-certs\" (UniqueName: \"kubernetes.io/secret/cc64ebb9-44e9-435a-9db0-d53ce5f139ff-tls-certs\") pod \"cc64ebb9-44e9-435a-9db0-d53ce5f139ff\" (UID: \"cc64ebb9-44e9-435a-9db0-d53ce5f139ff\") " Oct 30 19:23:47.151821 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:47.151252 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f51fd746-cd85-4608-96a6-f54afa0ec34d-model-cache" pod "f51fd746-cd85-4608-96a6-f54afa0ec34d" (UID: "f51fd746-cd85-4608-96a6-f54afa0ec34d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:23:47.151821 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:47.151490 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f51fd746-cd85-4608-96a6-f54afa0ec34d-tmp-dir" pod "f51fd746-cd85-4608-96a6-f54afa0ec34d" (UID: "f51fd746-cd85-4608-96a6-f54afa0ec34d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:23:47.151821 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:47.151779 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc64ebb9-44e9-435a-9db0-d53ce5f139ff-model-cache" pod "cc64ebb9-44e9-435a-9db0-d53ce5f139ff" (UID: "cc64ebb9-44e9-435a-9db0-d53ce5f139ff"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:23:47.152057 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:47.151976 6960 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc64ebb9-44e9-435a-9db0-d53ce5f139ff-tmp-dir\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:23:47.152057 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:47.151999 6960 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f51fd746-cd85-4608-96a6-f54afa0ec34d-tmp-dir\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:23:47.152057 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:47.152014 6960 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8701242-b59c-43b1-9c0a-011635edd419-tmp-dir\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:23:47.152057 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:47.152028 6960 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f51fd746-cd85-4608-96a6-f54afa0ec34d-model-cache\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:23:47.152057 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:47.152030 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc64ebb9-44e9-435a-9db0-d53ce5f139ff-home" pod "cc64ebb9-44e9-435a-9db0-d53ce5f139ff" (UID: "cc64ebb9-44e9-435a-9db0-d53ce5f139ff"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:23:47.152057 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:47.152042 6960 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8701242-b59c-43b1-9c0a-011635edd419-model-cache\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:23:47.152874 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:47.152832 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8701242-b59c-43b1-9c0a-011635edd419-home" pod "e8701242-b59c-43b1-9c0a-011635edd419" (UID: "e8701242-b59c-43b1-9c0a-011635edd419"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:23:47.153240 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:47.153215 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f51fd746-cd85-4608-96a6-f54afa0ec34d-home" pod "f51fd746-cd85-4608-96a6-f54afa0ec34d" (UID: "f51fd746-cd85-4608-96a6-f54afa0ec34d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:23:47.153720 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:47.153691 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f51fd746-cd85-4608-96a6-f54afa0ec34d-dshm" pod "f51fd746-cd85-4608-96a6-f54afa0ec34d" (UID: "f51fd746-cd85-4608-96a6-f54afa0ec34d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:23:47.154254 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:47.154226 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8701242-b59c-43b1-9c0a-011635edd419-kube-api-access-r822d" pod "e8701242-b59c-43b1-9c0a-011635edd419" (UID: "e8701242-b59c-43b1-9c0a-011635edd419"). InnerVolumeSpecName "kube-api-access-r822d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:23:47.154415 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:47.154387 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f51fd746-cd85-4608-96a6-f54afa0ec34d-tls-certs" pod "f51fd746-cd85-4608-96a6-f54afa0ec34d" (UID: "f51fd746-cd85-4608-96a6-f54afa0ec34d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:23:47.154489 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:47.154412 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc64ebb9-44e9-435a-9db0-d53ce5f139ff-kube-api-access-lg4ll" pod "cc64ebb9-44e9-435a-9db0-d53ce5f139ff" (UID: "cc64ebb9-44e9-435a-9db0-d53ce5f139ff"). InnerVolumeSpecName "kube-api-access-lg4ll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:23:47.154678 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:47.154652 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc64ebb9-44e9-435a-9db0-d53ce5f139ff-dshm" pod "cc64ebb9-44e9-435a-9db0-d53ce5f139ff" (UID: "cc64ebb9-44e9-435a-9db0-d53ce5f139ff"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:23:47.155742 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:47.155722 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8701242-b59c-43b1-9c0a-011635edd419-tls-certs" pod "e8701242-b59c-43b1-9c0a-011635edd419" (UID: "e8701242-b59c-43b1-9c0a-011635edd419"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:23:47.156239 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:47.156223 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc64ebb9-44e9-435a-9db0-d53ce5f139ff-tls-certs" pod "cc64ebb9-44e9-435a-9db0-d53ce5f139ff" (UID: "cc64ebb9-44e9-435a-9db0-d53ce5f139ff"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:23:47.156379 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:47.156361 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8701242-b59c-43b1-9c0a-011635edd419-dshm" pod "e8701242-b59c-43b1-9c0a-011635edd419" (UID: "e8701242-b59c-43b1-9c0a-011635edd419"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:23:47.157223 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:47.157195 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f51fd746-cd85-4608-96a6-f54afa0ec34d-kube-api-access-kl582" pod "f51fd746-cd85-4608-96a6-f54afa0ec34d" (UID: "f51fd746-cd85-4608-96a6-f54afa0ec34d"). InnerVolumeSpecName "kube-api-access-kl582". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:23:47.253514 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:47.253431 6960 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8701242-b59c-43b1-9c0a-011635edd419-home\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:23:47.253514 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:47.253462 6960 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8701242-b59c-43b1-9c0a-011635edd419-tls-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:23:47.253514 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:47.253474 6960 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc64ebb9-44e9-435a-9db0-d53ce5f139ff-home\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:23:47.253514 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:47.253486 6960 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc64ebb9-44e9-435a-9db0-d53ce5f139ff-dshm\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:23:47.253514 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:47.253497 6960 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f51fd746-cd85-4608-96a6-f54afa0ec34d-home\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:23:47.253514 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:47.253508 6960 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc64ebb9-44e9-435a-9db0-d53ce5f139ff-tls-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:23:47.253842 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:47.253522 6960 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f51fd746-cd85-4608-96a6-f54afa0ec34d-dshm\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:23:47.253842 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:47.253534 6960 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f51fd746-cd85-4608-96a6-f54afa0ec34d-tls-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:23:47.253842 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:47.253546 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lg4ll\" (UniqueName: \"kubernetes.io/projected/cc64ebb9-44e9-435a-9db0-d53ce5f139ff-kube-api-access-lg4ll\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:23:47.253842 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:47.253559 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kl582\" (UniqueName: \"kubernetes.io/projected/f51fd746-cd85-4608-96a6-f54afa0ec34d-kube-api-access-kl582\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:23:47.253842 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:47.253571 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r822d\" (UniqueName: \"kubernetes.io/projected/e8701242-b59c-43b1-9c0a-011635edd419-kube-api-access-r822d\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:23:47.253842 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:47.253583 6960 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc64ebb9-44e9-435a-9db0-d53ce5f139ff-model-cache\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:23:47.253842 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:47.253594 6960 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8701242-b59c-43b1-9c0a-011635edd419-dshm\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:23:47.331540 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:47.331504 6960 generic.go:356] "Generic (PLEG): container finished" podID="e8701242-b59c-43b1-9c0a-011635edd419" containerID="da8c7e67cc7d7dbc66bb234713ee5a5e9a629cf12bd94737827fc249f1996cce" exitCode=0 Oct 30 19:23:47.331711 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:47.331570 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-2-w3-7a8fa2-kserve-677bb44d8d-fg55k" Oct 30 19:23:47.331711 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:47.331586 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-2-w3-7a8fa2-kserve-677bb44d8d-fg55k" event={"ID":"e8701242-b59c-43b1-9c0a-011635edd419","Type":"ContainerDied","Data":"da8c7e67cc7d7dbc66bb234713ee5a5e9a629cf12bd94737827fc249f1996cce"} Oct 30 19:23:47.331711 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:47.331637 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-2-w3-7a8fa2-kserve-677bb44d8d-fg55k" event={"ID":"e8701242-b59c-43b1-9c0a-011635edd419","Type":"ContainerDied","Data":"0a8181a958839877f9bea10b17db1ab5ff7f14819ee0ec5d7b01f16ccf8594fe"} Oct 30 19:23:47.331711 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:47.331658 6960 scope.go:122] "RemoveContainer" containerID="da8c7e67cc7d7dbc66bb234713ee5a5e9a629cf12bd94737827fc249f1996cce" Oct 30 19:23:47.333088 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:47.333032 6960 generic.go:356] "Generic (PLEG): container finished" podID="cc64ebb9-44e9-435a-9db0-d53ce5f139ff" containerID="1971b54e599f3ec447c9fc92d5fa8bde1edea9aebb77a9805e45805e6765d9d7" exitCode=0 Oct 30 19:23:47.333088 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:47.333069 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-7a8fa2/simulator-w3-7a8fa2-kserve-78587cb8fc-6pvsz" Oct 30 19:23:47.333282 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:47.333115 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-7a8fa2/simulator-w3-7a8fa2-kserve-78587cb8fc-6pvsz" event={"ID":"cc64ebb9-44e9-435a-9db0-d53ce5f139ff","Type":"ContainerDied","Data":"1971b54e599f3ec447c9fc92d5fa8bde1edea9aebb77a9805e45805e6765d9d7"} Oct 30 19:23:47.333282 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:47.333166 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-7a8fa2/simulator-w3-7a8fa2-kserve-78587cb8fc-6pvsz" event={"ID":"cc64ebb9-44e9-435a-9db0-d53ce5f139ff","Type":"ContainerDied","Data":"1cf18d107341fbfd9bf8a254030a6afa1d5c4b0e014cee0e391f2cc909e522e5"} Oct 30 19:23:47.334949 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:47.334909 6960 generic.go:356] "Generic (PLEG): container finished" podID="f51fd746-cd85-4608-96a6-f54afa0ec34d" containerID="7272ada118e39fc2b5c37624c620ca1ffdf53ff85c9374e52e94e678b28b5c98" exitCode=0 Oct 30 19:23:47.335079 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:47.335046 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-7a8fa2/premium-w3-7a8fa2-kserve-6b6c5f6fb8-n5wtq" event={"ID":"f51fd746-cd85-4608-96a6-f54afa0ec34d","Type":"ContainerDied","Data":"7272ada118e39fc2b5c37624c620ca1ffdf53ff85c9374e52e94e678b28b5c98"} Oct 30 19:23:47.335079 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:47.335075 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-7a8fa2/premium-w3-7a8fa2-kserve-6b6c5f6fb8-n5wtq" event={"ID":"f51fd746-cd85-4608-96a6-f54afa0ec34d","Type":"ContainerDied","Data":"48cb28d2f5bffe3d94dca03a91ab9c37beff32cb318f25b868b86f9d51a5cef2"} Oct 30 19:23:47.335257 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:47.335165 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-7a8fa2/premium-w3-7a8fa2-kserve-6b6c5f6fb8-n5wtq" Oct 30 19:23:47.344285 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:47.344267 6960 scope.go:122] "RemoveContainer" containerID="da8c7e67cc7d7dbc66bb234713ee5a5e9a629cf12bd94737827fc249f1996cce" Oct 30 19:23:47.344580 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:23:47.344550 6960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da8c7e67cc7d7dbc66bb234713ee5a5e9a629cf12bd94737827fc249f1996cce\": container with ID starting with da8c7e67cc7d7dbc66bb234713ee5a5e9a629cf12bd94737827fc249f1996cce not found: ID does not exist" containerID="da8c7e67cc7d7dbc66bb234713ee5a5e9a629cf12bd94737827fc249f1996cce" Oct 30 19:23:47.344642 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:47.344588 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da8c7e67cc7d7dbc66bb234713ee5a5e9a629cf12bd94737827fc249f1996cce"} err="failed to get container status \"da8c7e67cc7d7dbc66bb234713ee5a5e9a629cf12bd94737827fc249f1996cce\": rpc error: code = NotFound desc = could not find container \"da8c7e67cc7d7dbc66bb234713ee5a5e9a629cf12bd94737827fc249f1996cce\": container with ID starting with da8c7e67cc7d7dbc66bb234713ee5a5e9a629cf12bd94737827fc249f1996cce not found: ID does not exist" Oct 30 19:23:47.344642 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:47.344610 6960 scope.go:122] "RemoveContainer" containerID="1971b54e599f3ec447c9fc92d5fa8bde1edea9aebb77a9805e45805e6765d9d7" Oct 30 19:23:47.353724 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:47.353705 6960 scope.go:122] "RemoveContainer" containerID="1971b54e599f3ec447c9fc92d5fa8bde1edea9aebb77a9805e45805e6765d9d7" Oct 30 19:23:47.354006 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:23:47.353989 6960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1971b54e599f3ec447c9fc92d5fa8bde1edea9aebb77a9805e45805e6765d9d7\": container with ID starting with 1971b54e599f3ec447c9fc92d5fa8bde1edea9aebb77a9805e45805e6765d9d7 not found: ID does not exist" containerID="1971b54e599f3ec447c9fc92d5fa8bde1edea9aebb77a9805e45805e6765d9d7" Oct 30 19:23:47.354061 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:47.354013 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1971b54e599f3ec447c9fc92d5fa8bde1edea9aebb77a9805e45805e6765d9d7"} err="failed to get container status \"1971b54e599f3ec447c9fc92d5fa8bde1edea9aebb77a9805e45805e6765d9d7\": rpc error: code = NotFound desc = could not find container \"1971b54e599f3ec447c9fc92d5fa8bde1edea9aebb77a9805e45805e6765d9d7\": container with ID starting with 1971b54e599f3ec447c9fc92d5fa8bde1edea9aebb77a9805e45805e6765d9d7 not found: ID does not exist" Oct 30 19:23:47.354061 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:47.354031 6960 scope.go:122] "RemoveContainer" containerID="7272ada118e39fc2b5c37624c620ca1ffdf53ff85c9374e52e94e678b28b5c98" Oct 30 19:23:47.360900 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:47.360870 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-7a8fa2/simulator-w3-7a8fa2-kserve-78587cb8fc-6pvsz"] Oct 30 19:23:47.363395 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:47.363379 6960 scope.go:122] "RemoveContainer" containerID="7272ada118e39fc2b5c37624c620ca1ffdf53ff85c9374e52e94e678b28b5c98" Oct 30 19:23:47.363639 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:23:47.363622 6960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7272ada118e39fc2b5c37624c620ca1ffdf53ff85c9374e52e94e678b28b5c98\": container with ID starting with 7272ada118e39fc2b5c37624c620ca1ffdf53ff85c9374e52e94e678b28b5c98 not found: ID does not exist" containerID="7272ada118e39fc2b5c37624c620ca1ffdf53ff85c9374e52e94e678b28b5c98" Oct 30 19:23:47.363681 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:47.363648 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7272ada118e39fc2b5c37624c620ca1ffdf53ff85c9374e52e94e678b28b5c98"} err="failed to get container status \"7272ada118e39fc2b5c37624c620ca1ffdf53ff85c9374e52e94e678b28b5c98\": rpc error: code = NotFound desc = could not find container \"7272ada118e39fc2b5c37624c620ca1ffdf53ff85c9374e52e94e678b28b5c98\": container with ID starting with 7272ada118e39fc2b5c37624c620ca1ffdf53ff85c9374e52e94e678b28b5c98 not found: ID does not exist" Oct 30 19:23:47.372256 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:47.372235 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-7a8fa2/simulator-w3-7a8fa2-kserve-78587cb8fc-6pvsz"] Oct 30 19:23:47.384368 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:47.384349 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-7a8fa2/distinct-2-w3-7a8fa2-kserve-677bb44d8d-fg55k"] Oct 30 19:23:47.390108 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:47.390083 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-7a8fa2/distinct-2-w3-7a8fa2-kserve-677bb44d8d-fg55k"] Oct 30 19:23:47.402099 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:47.402079 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-7a8fa2/premium-w3-7a8fa2-kserve-6b6c5f6fb8-n5wtq"] Oct 30 19:23:47.414103 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:47.414083 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-7a8fa2/premium-w3-7a8fa2-kserve-6b6c5f6fb8-n5wtq"] Oct 30 19:23:47.824315 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:47.824283 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:47.949884 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:47.949805 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-7a8fa2/distinct-2-w3-7a8fa2-kserve-677bb44d8d-fg55k" podUID="e8701242-b59c-43b1-9c0a-011635edd419" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:23:48.297014 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:48.296932 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-7a8fa2-778fb96bb-tlvqh" podUID="67a3f38a-10e8-4826-925c-520ce8297f11" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:23:48.593691 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:48.593657 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc64ebb9-44e9-435a-9db0-d53ce5f139ff" path="/var/lib/kubelet/pods/cc64ebb9-44e9-435a-9db0-d53ce5f139ff/volumes" Oct 30 19:23:48.594095 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:48.594080 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8701242-b59c-43b1-9c0a-011635edd419" path="/var/lib/kubelet/pods/e8701242-b59c-43b1-9c0a-011635edd419/volumes" Oct 30 19:23:48.594500 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:48.594487 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f51fd746-cd85-4608-96a6-f54afa0ec34d" path="/var/lib/kubelet/pods/f51fd746-cd85-4608-96a6-f54afa0ec34d/volumes" Oct 30 19:23:48.610403 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:48.610372 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:49.435423 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:49.435391 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:50.328095 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:50.328057 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:51.480368 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:51.480325 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:51.941446 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:51.941389 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:52.870857 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:52.870829 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:53.857431 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:53.857402 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:53.932316 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:23:53.932279 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 19:23:54.714971 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:54.714934 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:54.840455 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:54.840416 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:55.603545 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:55.603504 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:56.475746 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:56.475712 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:57.323439 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:57.323393 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:57.931129 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:57.931084 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-7a8fa2-59479c7948-wtw6n"] Oct 30 19:23:57.931372 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:57.931348 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7a8fa2-59479c7948-wtw6n" podUID="da7fcb8d-db2d-4b43-a04f-22f4a9c0757b" containerName="maas-api" containerID="cri-o://4e52eb5020a77f2e16766747cc8c791fe7c159b0e3542ce3450bac64cf367ff0" gracePeriod=30 Oct 30 19:23:57.995201 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:57.995170 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:58.209849 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:58.209734 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:58.259500 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:58.259470 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7a8fa2-59479c7948-wtw6n" Oct 30 19:23:58.296749 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:58.296714 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-7a8fa2-778fb96bb-tlvqh" podUID="67a3f38a-10e8-4826-925c-520ce8297f11" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:23:58.380657 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:58.380612 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:58.400544 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:58.400516 6960 generic.go:356] "Generic (PLEG): container finished" podID="da7fcb8d-db2d-4b43-a04f-22f4a9c0757b" containerID="4e52eb5020a77f2e16766747cc8c791fe7c159b0e3542ce3450bac64cf367ff0" exitCode=0 Oct 30 19:23:58.400745 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:58.400562 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7a8fa2-59479c7948-wtw6n" Oct 30 19:23:58.400745 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:58.400596 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7a8fa2-59479c7948-wtw6n" event={"ID":"da7fcb8d-db2d-4b43-a04f-22f4a9c0757b","Type":"ContainerDied","Data":"4e52eb5020a77f2e16766747cc8c791fe7c159b0e3542ce3450bac64cf367ff0"} Oct 30 19:23:58.400745 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:58.400663 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7a8fa2-59479c7948-wtw6n" event={"ID":"da7fcb8d-db2d-4b43-a04f-22f4a9c0757b","Type":"ContainerDied","Data":"49e8c29c25fde0b0e69e5c701421ed553dde3ea1e6a7c3b1329ca854fabdbc4a"} Oct 30 19:23:58.400745 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:58.400697 6960 scope.go:122] "RemoveContainer" containerID="4e52eb5020a77f2e16766747cc8c791fe7c159b0e3542ce3450bac64cf367ff0" Oct 30 19:23:58.408223 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:58.408201 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da7fcb8d-db2d-4b43-a04f-22f4a9c0757b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da7fcb8d-db2d-4b43-a04f-22f4a9c0757b-openshift-service-ca\") pod \"da7fcb8d-db2d-4b43-a04f-22f4a9c0757b\" (UID: \"da7fcb8d-db2d-4b43-a04f-22f4a9c0757b\") " Oct 30 19:23:58.408533 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:58.408304 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da7fcb8d-db2d-4b43-a04f-22f4a9c0757b-kube-api-access-9vrgn\" (UniqueName: \"kubernetes.io/projected/da7fcb8d-db2d-4b43-a04f-22f4a9c0757b-kube-api-access-9vrgn\") pod \"da7fcb8d-db2d-4b43-a04f-22f4a9c0757b\" (UID: \"da7fcb8d-db2d-4b43-a04f-22f4a9c0757b\") " Oct 30 19:23:58.408533 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:58.408371 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da7fcb8d-db2d-4b43-a04f-22f4a9c0757b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da7fcb8d-db2d-4b43-a04f-22f4a9c0757b-maas-api-tls\") pod \"da7fcb8d-db2d-4b43-a04f-22f4a9c0757b\" (UID: \"da7fcb8d-db2d-4b43-a04f-22f4a9c0757b\") " Oct 30 19:23:58.408778 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:58.408592 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da7fcb8d-db2d-4b43-a04f-22f4a9c0757b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/da7fcb8d-db2d-4b43-a04f-22f4a9c0757b-metrics-certs\") pod \"da7fcb8d-db2d-4b43-a04f-22f4a9c0757b\" (UID: \"da7fcb8d-db2d-4b43-a04f-22f4a9c0757b\") " Oct 30 19:23:58.408778 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:58.408605 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da7fcb8d-db2d-4b43-a04f-22f4a9c0757b-openshift-service-ca" pod "da7fcb8d-db2d-4b43-a04f-22f4a9c0757b" (UID: "da7fcb8d-db2d-4b43-a04f-22f4a9c0757b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:23:58.409160 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:58.409122 6960 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da7fcb8d-db2d-4b43-a04f-22f4a9c0757b-openshift-service-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:23:58.410785 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:58.410758 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da7fcb8d-db2d-4b43-a04f-22f4a9c0757b-maas-api-tls" pod "da7fcb8d-db2d-4b43-a04f-22f4a9c0757b" (UID: "da7fcb8d-db2d-4b43-a04f-22f4a9c0757b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:23:58.410993 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:58.410967 6960 scope.go:122] "RemoveContainer" containerID="4e52eb5020a77f2e16766747cc8c791fe7c159b0e3542ce3450bac64cf367ff0" Oct 30 19:23:58.410993 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:58.410978 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da7fcb8d-db2d-4b43-a04f-22f4a9c0757b-kube-api-access-9vrgn" pod "da7fcb8d-db2d-4b43-a04f-22f4a9c0757b" (UID: "da7fcb8d-db2d-4b43-a04f-22f4a9c0757b"). InnerVolumeSpecName "kube-api-access-9vrgn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:23:58.411198 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:58.411000 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da7fcb8d-db2d-4b43-a04f-22f4a9c0757b-metrics-certs" pod "da7fcb8d-db2d-4b43-a04f-22f4a9c0757b" (UID: "da7fcb8d-db2d-4b43-a04f-22f4a9c0757b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:23:58.411343 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:23:58.411321 6960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e52eb5020a77f2e16766747cc8c791fe7c159b0e3542ce3450bac64cf367ff0\": container with ID starting with 4e52eb5020a77f2e16766747cc8c791fe7c159b0e3542ce3450bac64cf367ff0 not found: ID does not exist" containerID="4e52eb5020a77f2e16766747cc8c791fe7c159b0e3542ce3450bac64cf367ff0" Oct 30 19:23:58.411407 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:58.411348 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e52eb5020a77f2e16766747cc8c791fe7c159b0e3542ce3450bac64cf367ff0"} err="failed to get container status \"4e52eb5020a77f2e16766747cc8c791fe7c159b0e3542ce3450bac64cf367ff0\": rpc error: code = NotFound desc = could not find container \"4e52eb5020a77f2e16766747cc8c791fe7c159b0e3542ce3450bac64cf367ff0\": container with ID starting with 4e52eb5020a77f2e16766747cc8c791fe7c159b0e3542ce3450bac64cf367ff0 not found: ID does not exist" Oct 30 19:23:58.510196 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:58.510095 6960 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da7fcb8d-db2d-4b43-a04f-22f4a9c0757b-maas-api-tls\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:23:58.510196 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:58.510120 6960 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da7fcb8d-db2d-4b43-a04f-22f4a9c0757b-metrics-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:23:58.510196 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:58.510129 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vrgn\" (UniqueName: \"kubernetes.io/projected/da7fcb8d-db2d-4b43-a04f-22f4a9c0757b-kube-api-access-9vrgn\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:23:58.636524 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:58.636493 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:58.719892 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:58.719845 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-7a8fa2-59479c7948-wtw6n"] Oct 30 19:23:58.723903 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:58.723876 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-7a8fa2-59479c7948-wtw6n"] Oct 30 19:23:58.870769 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:58.870741 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:23:59.405460 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:23:59.405413 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:00.312553 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:00.312512 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:00.592490 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:00.592409 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da7fcb8d-db2d-4b43-a04f-22f4a9c0757b" path="/var/lib/kubelet/pods/da7fcb8d-db2d-4b43-a04f-22f4a9c0757b/volumes" Oct 30 19:24:01.098382 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:01.098325 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:01.846250 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:01.846194 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:02.220898 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:02.220867 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-7a8fa2-778fb96bb-tlvqh" Oct 30 19:24:02.254067 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:02.254039 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67a3f38a-10e8-4826-925c-520ce8297f11-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67a3f38a-10e8-4826-925c-520ce8297f11-plugins-config-volume\") pod \"67a3f38a-10e8-4826-925c-520ce8297f11\" (UID: \"67a3f38a-10e8-4826-925c-520ce8297f11\") " Oct 30 19:24:02.254286 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:02.254186 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67a3f38a-10e8-4826-925c-520ce8297f11-kube-api-access-7x7k5\" (UniqueName: \"kubernetes.io/projected/67a3f38a-10e8-4826-925c-520ce8297f11-kube-api-access-7x7k5\") pod \"67a3f38a-10e8-4826-925c-520ce8297f11\" (UID: \"67a3f38a-10e8-4826-925c-520ce8297f11\") " Oct 30 19:24:02.254763 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:02.254460 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67a3f38a-10e8-4826-925c-520ce8297f11-plugins-config-volume" pod "67a3f38a-10e8-4826-925c-520ce8297f11" (UID: "67a3f38a-10e8-4826-925c-520ce8297f11"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:24:02.254763 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:02.254583 6960 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67a3f38a-10e8-4826-925c-520ce8297f11-plugins-config-volume\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:24:02.256765 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:02.256738 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67a3f38a-10e8-4826-925c-520ce8297f11-kube-api-access-7x7k5" pod "67a3f38a-10e8-4826-925c-520ce8297f11" (UID: "67a3f38a-10e8-4826-925c-520ce8297f11"). InnerVolumeSpecName "kube-api-access-7x7k5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:24:02.355220 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:02.355186 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7x7k5\" (UniqueName: \"kubernetes.io/projected/67a3f38a-10e8-4826-925c-520ce8297f11-kube-api-access-7x7k5\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:24:02.425548 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:02.425466 6960 generic.go:356] "Generic (PLEG): container finished" podID="67a3f38a-10e8-4826-925c-520ce8297f11" containerID="e5d7327234db62163358bdb4e01d1231e9e3d2ea60891475b591eed63452145f" exitCode=0 Oct 30 19:24:02.425548 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:02.425510 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-7a8fa2-778fb96bb-tlvqh" event={"ID":"67a3f38a-10e8-4826-925c-520ce8297f11","Type":"ContainerDied","Data":"e5d7327234db62163358bdb4e01d1231e9e3d2ea60891475b591eed63452145f"} Oct 30 19:24:02.425548 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:02.425522 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-7a8fa2-778fb96bb-tlvqh" Oct 30 19:24:02.425814 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:02.425575 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-7a8fa2-778fb96bb-tlvqh" event={"ID":"67a3f38a-10e8-4826-925c-520ce8297f11","Type":"ContainerDied","Data":"37ca11a25b2fd43b339f3f7101d52aeac9be645b18d60129f36f0eb93b6b7d0a"} Oct 30 19:24:02.425814 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:02.425596 6960 scope.go:122] "RemoveContainer" containerID="e5d7327234db62163358bdb4e01d1231e9e3d2ea60891475b591eed63452145f" Oct 30 19:24:02.435434 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:02.435414 6960 scope.go:122] "RemoveContainer" containerID="e5d7327234db62163358bdb4e01d1231e9e3d2ea60891475b591eed63452145f" Oct 30 19:24:02.435699 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:24:02.435681 6960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5d7327234db62163358bdb4e01d1231e9e3d2ea60891475b591eed63452145f\": container with ID starting with e5d7327234db62163358bdb4e01d1231e9e3d2ea60891475b591eed63452145f not found: ID does not exist" containerID="e5d7327234db62163358bdb4e01d1231e9e3d2ea60891475b591eed63452145f" Oct 30 19:24:02.435753 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:02.435711 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5d7327234db62163358bdb4e01d1231e9e3d2ea60891475b591eed63452145f"} err="failed to get container status \"e5d7327234db62163358bdb4e01d1231e9e3d2ea60891475b591eed63452145f\": rpc error: code = NotFound desc = could not find container \"e5d7327234db62163358bdb4e01d1231e9e3d2ea60891475b591eed63452145f\": container with ID starting with e5d7327234db62163358bdb4e01d1231e9e3d2ea60891475b591eed63452145f not found: ID does not exist" Oct 30 19:24:02.451435 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:02.451408 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-7a8fa2-778fb96bb-tlvqh"] Oct 30 19:24:02.451542 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:02.451449 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-7a8fa2-778fb96bb-tlvqh"] Oct 30 19:24:02.593103 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:02.593071 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67a3f38a-10e8-4826-925c-520ce8297f11" path="/var/lib/kubelet/pods/67a3f38a-10e8-4826-925c-520ce8297f11/volumes" Oct 30 19:24:02.734331 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:02.734252 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:03.518518 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:03.518483 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:04.600073 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:04.600036 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:05.575901 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:05.575871 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:06.348465 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:06.348424 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:06.461925 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:06.461886 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:06.704393 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:06.704306 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-phklq"] Oct 30 19:24:06.704598 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:06.704570 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-phklq" podUID="67bc209e-b148-4464-912e-f1ba5f372c50" containerName="istio-proxy" containerID="cri-o://81fe2a9bc4b5feb43b828bfa531e8b17f3aa84c0e47f4e428856ea2a229749ec" gracePeriod=30 Oct 30 19:24:06.750956 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:06.750903 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:07.500596 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:07.500556 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:08.228266 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:08.228234 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:08.376689 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:08.376654 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:08.508622 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:08.508523 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:08.625004 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:08.624962 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:08.754608 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:08.754571 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:09.495535 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:09.495493 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:10.490844 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:10.490811 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:11.438470 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:11.438432 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:11.564439 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:11.564405 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:12.009568 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:12.009543 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-phklq" Oct 30 19:24:12.089569 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:12.089535 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67bc209e-b148-4464-912e-f1ba5f372c50-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67bc209e-b148-4464-912e-f1ba5f372c50-istio-envoy\") pod \"67bc209e-b148-4464-912e-f1ba5f372c50\" (UID: \"67bc209e-b148-4464-912e-f1ba5f372c50\") " Oct 30 19:24:12.089759 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:12.089652 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67bc209e-b148-4464-912e-f1ba5f372c50-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67bc209e-b148-4464-912e-f1ba5f372c50-istiod-ca-cert\") pod \"67bc209e-b148-4464-912e-f1ba5f372c50\" (UID: \"67bc209e-b148-4464-912e-f1ba5f372c50\") " Oct 30 19:24:12.089759 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:12.089715 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67bc209e-b148-4464-912e-f1ba5f372c50-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67bc209e-b148-4464-912e-f1ba5f372c50-credential-socket\") pod \"67bc209e-b148-4464-912e-f1ba5f372c50\" (UID: \"67bc209e-b148-4464-912e-f1ba5f372c50\") " Oct 30 19:24:12.089886 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:12.089796 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67bc209e-b148-4464-912e-f1ba5f372c50-kube-api-access-pljbr\" (UniqueName: \"kubernetes.io/projected/67bc209e-b148-4464-912e-f1ba5f372c50-kube-api-access-pljbr\") pod \"67bc209e-b148-4464-912e-f1ba5f372c50\" (UID: \"67bc209e-b148-4464-912e-f1ba5f372c50\") " Oct 30 19:24:12.089886 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:12.089828 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67bc209e-b148-4464-912e-f1ba5f372c50-istio-token\" (UniqueName: \"kubernetes.io/projected/67bc209e-b148-4464-912e-f1ba5f372c50-istio-token\") pod \"67bc209e-b148-4464-912e-f1ba5f372c50\" (UID: \"67bc209e-b148-4464-912e-f1ba5f372c50\") " Oct 30 19:24:12.089886 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:12.089875 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67bc209e-b148-4464-912e-f1ba5f372c50-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67bc209e-b148-4464-912e-f1ba5f372c50-workload-socket\") pod \"67bc209e-b148-4464-912e-f1ba5f372c50\" (UID: \"67bc209e-b148-4464-912e-f1ba5f372c50\") " Oct 30 19:24:12.090039 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:12.089903 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67bc209e-b148-4464-912e-f1ba5f372c50-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67bc209e-b148-4464-912e-f1ba5f372c50-workload-certs\") pod \"67bc209e-b148-4464-912e-f1ba5f372c50\" (UID: \"67bc209e-b148-4464-912e-f1ba5f372c50\") " Oct 30 19:24:12.090039 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:12.089954 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67bc209e-b148-4464-912e-f1ba5f372c50-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67bc209e-b148-4464-912e-f1ba5f372c50-istio-data\") pod \"67bc209e-b148-4464-912e-f1ba5f372c50\" (UID: \"67bc209e-b148-4464-912e-f1ba5f372c50\") " Oct 30 19:24:12.090039 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:12.089988 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/67bc209e-b148-4464-912e-f1ba5f372c50-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67bc209e-b148-4464-912e-f1ba5f372c50-istio-podinfo\") pod \"67bc209e-b148-4464-912e-f1ba5f372c50\" (UID: \"67bc209e-b148-4464-912e-f1ba5f372c50\") " Oct 30 19:24:12.090243 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:12.090077 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67bc209e-b148-4464-912e-f1ba5f372c50-credential-socket" pod "67bc209e-b148-4464-912e-f1ba5f372c50" (UID: "67bc209e-b148-4464-912e-f1ba5f372c50"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:24:12.090243 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:12.090082 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67bc209e-b148-4464-912e-f1ba5f372c50-istiod-ca-cert" pod "67bc209e-b148-4464-912e-f1ba5f372c50" (UID: "67bc209e-b148-4464-912e-f1ba5f372c50"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:24:12.090387 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:12.090365 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67bc209e-b148-4464-912e-f1ba5f372c50-istio-data" pod "67bc209e-b148-4464-912e-f1ba5f372c50" (UID: "67bc209e-b148-4464-912e-f1ba5f372c50"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:24:12.090451 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:12.090428 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67bc209e-b148-4464-912e-f1ba5f372c50-workload-certs" pod "67bc209e-b148-4464-912e-f1ba5f372c50" (UID: "67bc209e-b148-4464-912e-f1ba5f372c50"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:24:12.090537 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:12.090380 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67bc209e-b148-4464-912e-f1ba5f372c50-workload-socket" pod "67bc209e-b148-4464-912e-f1ba5f372c50" (UID: "67bc209e-b148-4464-912e-f1ba5f372c50"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:24:12.090594 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:12.090579 6960 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67bc209e-b148-4464-912e-f1ba5f372c50-workload-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:24:12.090653 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:12.090594 6960 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67bc209e-b148-4464-912e-f1ba5f372c50-istio-data\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:24:12.090653 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:12.090605 6960 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67bc209e-b148-4464-912e-f1ba5f372c50-istiod-ca-cert\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:24:12.090653 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:12.090615 6960 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67bc209e-b148-4464-912e-f1ba5f372c50-credential-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:24:12.092307 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:12.092281 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67bc209e-b148-4464-912e-f1ba5f372c50-istio-envoy" pod "67bc209e-b148-4464-912e-f1ba5f372c50" (UID: "67bc209e-b148-4464-912e-f1ba5f372c50"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:24:12.092612 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:12.092585 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67bc209e-b148-4464-912e-f1ba5f372c50-istio-token" pod "67bc209e-b148-4464-912e-f1ba5f372c50" (UID: "67bc209e-b148-4464-912e-f1ba5f372c50"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:24:12.092689 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:12.092624 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/67bc209e-b148-4464-912e-f1ba5f372c50-istio-podinfo" pod "67bc209e-b148-4464-912e-f1ba5f372c50" (UID: "67bc209e-b148-4464-912e-f1ba5f372c50"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:24:12.092824 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:12.092803 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67bc209e-b148-4464-912e-f1ba5f372c50-kube-api-access-pljbr" pod "67bc209e-b148-4464-912e-f1ba5f372c50" (UID: "67bc209e-b148-4464-912e-f1ba5f372c50"). InnerVolumeSpecName "kube-api-access-pljbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:24:12.192537 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:12.192503 6960 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67bc209e-b148-4464-912e-f1ba5f372c50-istio-envoy\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:24:12.192537 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:12.192532 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pljbr\" (UniqueName: \"kubernetes.io/projected/67bc209e-b148-4464-912e-f1ba5f372c50-kube-api-access-pljbr\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:24:12.192537 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:12.192543 6960 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67bc209e-b148-4464-912e-f1ba5f372c50-istio-token\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:24:12.192773 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:12.192555 6960 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67bc209e-b148-4464-912e-f1ba5f372c50-workload-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:24:12.192773 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:12.192570 6960 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67bc209e-b148-4464-912e-f1ba5f372c50-istio-podinfo\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:24:12.422535 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:12.422497 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:12.504767 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:12.504730 6960 generic.go:356] "Generic (PLEG): container finished" podID="67bc209e-b148-4464-912e-f1ba5f372c50" containerID="81fe2a9bc4b5feb43b828bfa531e8b17f3aa84c0e47f4e428856ea2a229749ec" exitCode=0 Oct 30 19:24:12.504959 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:12.504777 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-phklq" event={"ID":"67bc209e-b148-4464-912e-f1ba5f372c50","Type":"ContainerDied","Data":"81fe2a9bc4b5feb43b828bfa531e8b17f3aa84c0e47f4e428856ea2a229749ec"} Oct 30 19:24:12.504959 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:12.504821 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-phklq" event={"ID":"67bc209e-b148-4464-912e-f1ba5f372c50","Type":"ContainerDied","Data":"a1c2286ea525632daa3dc36f44c93ac68cdcc5d0d73a13476797e8da7fbd13c3"} Oct 30 19:24:12.504959 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:12.504841 6960 scope.go:122] "RemoveContainer" containerID="81fe2a9bc4b5feb43b828bfa531e8b17f3aa84c0e47f4e428856ea2a229749ec" Oct 30 19:24:12.504959 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:12.504785 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-phklq" Oct 30 19:24:12.520002 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:12.519985 6960 scope.go:122] "RemoveContainer" containerID="81fe2a9bc4b5feb43b828bfa531e8b17f3aa84c0e47f4e428856ea2a229749ec" Oct 30 19:24:12.520367 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:24:12.520339 6960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81fe2a9bc4b5feb43b828bfa531e8b17f3aa84c0e47f4e428856ea2a229749ec\": container with ID starting with 81fe2a9bc4b5feb43b828bfa531e8b17f3aa84c0e47f4e428856ea2a229749ec not found: ID does not exist" containerID="81fe2a9bc4b5feb43b828bfa531e8b17f3aa84c0e47f4e428856ea2a229749ec" Oct 30 19:24:12.520458 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:12.520403 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81fe2a9bc4b5feb43b828bfa531e8b17f3aa84c0e47f4e428856ea2a229749ec"} err="failed to get container status \"81fe2a9bc4b5feb43b828bfa531e8b17f3aa84c0e47f4e428856ea2a229749ec\": rpc error: code = NotFound desc = could not find container \"81fe2a9bc4b5feb43b828bfa531e8b17f3aa84c0e47f4e428856ea2a229749ec\": container with ID starting with 81fe2a9bc4b5feb43b828bfa531e8b17f3aa84c0e47f4e428856ea2a229749ec not found: ID does not exist" Oct 30 19:24:12.529816 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:12.529792 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-phklq"] Oct 30 19:24:12.537375 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:12.537353 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-7a8fa2-openshift-default-798c9b87b9-phklq"] Oct 30 19:24:12.600913 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:12.600879 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67bc209e-b148-4464-912e-f1ba5f372c50" path="/var/lib/kubelet/pods/67bc209e-b148-4464-912e-f1ba5f372c50/volumes" Oct 30 19:24:13.265068 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:13.265032 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:13.560397 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:13.560367 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:14.057662 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:14.057627 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:14.921459 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:14.921403 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:15.711819 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:15.711786 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:16.592822 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:16.592787 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:17.508612 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:17.508574 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:18.449209 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:18.449173 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:18.900253 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:18.900225 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:19.304450 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:19.304380 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:20.207163 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:20.207118 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:21.030818 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:21.030788 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:21.883336 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:21.883295 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:22.729000 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:22.728964 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:23.513197 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:23.513125 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:23.932846 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:24:23.932814 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 19:24:24.301797 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:24.301724 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:25.101811 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:25.101768 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:26.118667 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:26.118626 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:26.919041 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:26.919006 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:27.907700 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:27.907657 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:28.713955 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:28.713917 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:29.517923 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:29.517886 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:30.301979 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:30.301947 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:31.110404 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:31.110364 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:31.906422 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:31.906388 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:32.710918 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:32.710877 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:33.701052 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:33.701021 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:34.519523 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:34.519488 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:35.042294 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:35.042254 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:35.170653 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:35.170614 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:35.316315 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:35.316283 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:36.244218 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:36.244126 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:37.123681 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:37.123645 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:37.394825 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:37.394746 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:38.034009 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:38.033978 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:39.018112 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:39.018071 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:39.817031 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:39.816995 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:40.822302 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:40.822257 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:41.626265 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:41.626230 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:42.404261 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:42.404226 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:43.204512 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:43.204474 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:44.034160 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:44.034109 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:44.825881 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:44.825809 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:45.639890 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:45.639839 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:46.558653 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:46.558600 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:47.597419 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:47.597369 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:48.327061 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:48.327027 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:49.142900 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:49.142863 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:50.039170 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:50.027799 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:50.846297 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:50.846268 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:51.831529 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:51.831494 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:52.706587 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:52.706545 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:53.507497 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:53.504837 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:53.932266 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:24:53.932230 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 19:24:54.318702 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:54.318671 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:54.858275 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:54.858241 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:55.122246 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:55.122132 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:56.045689 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:56.045639 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:56.914584 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:56.914554 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:57.698200 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:57.698166 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:58.600103 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:58.600061 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:24:59.404902 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:24:59.404865 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:00.209175 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:00.209127 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:01.099125 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:01.099089 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:01.893543 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:01.893509 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:02.645237 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:02.645207 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:03.505051 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:03.504974 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:04.285114 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:04.285066 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:05.095577 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:05.095540 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:06.030940 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:06.030905 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:07.010831 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:07.010788 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:07.863801 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:07.863766 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:08.857126 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:08.857089 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:08.966805 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:08.966769 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:09.823246 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:09.823213 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:10.804455 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:10.804422 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:11.747649 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:11.747611 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:12.621816 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:12.621764 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:13.689022 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:13.688983 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:14.060497 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:14.060454 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:14.647088 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:14.647033 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:15.608667 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:15.608632 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:16.594610 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:16.594574 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:17.790974 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:17.790930 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:18.674126 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:18.674083 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:19.458059 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:19.458024 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:20.398973 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:20.398941 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:20.847469 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:20.847429 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zgl9b"] Oct 30 19:25:20.847969 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:20.847661 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zgl9b" podUID="9c25987f-3a86-4c2e-ae7c-41080e0a3e24" containerName="discovery" containerID="cri-o://ad3386ab527b9f935dbb792f3fb6567951172f5014c0e8b36b7a4b41629998ac" gracePeriod=30 Oct 30 19:25:21.106411 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:21.106357 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zgl9b" Oct 30 19:25:21.258902 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:21.258863 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:21.285270 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:21.285234 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c25987f-3a86-4c2e-ae7c-41080e0a3e24-istio-token\" (UniqueName: \"kubernetes.io/projected/9c25987f-3a86-4c2e-ae7c-41080e0a3e24-istio-token\") pod \"9c25987f-3a86-4c2e-ae7c-41080e0a3e24\" (UID: \"9c25987f-3a86-4c2e-ae7c-41080e0a3e24\") " Oct 30 19:25:21.285471 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:21.285298 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c25987f-3a86-4c2e-ae7c-41080e0a3e24-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9c25987f-3a86-4c2e-ae7c-41080e0a3e24-istio-csr-dns-cert\") pod \"9c25987f-3a86-4c2e-ae7c-41080e0a3e24\" (UID: \"9c25987f-3a86-4c2e-ae7c-41080e0a3e24\") " Oct 30 19:25:21.285471 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:21.285389 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c25987f-3a86-4c2e-ae7c-41080e0a3e24-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c25987f-3a86-4c2e-ae7c-41080e0a3e24-local-certs\") pod \"9c25987f-3a86-4c2e-ae7c-41080e0a3e24\" (UID: \"9c25987f-3a86-4c2e-ae7c-41080e0a3e24\") " Oct 30 19:25:21.285471 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:21.285425 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c25987f-3a86-4c2e-ae7c-41080e0a3e24-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9c25987f-3a86-4c2e-ae7c-41080e0a3e24-istio-csr-ca-configmap\") pod \"9c25987f-3a86-4c2e-ae7c-41080e0a3e24\" (UID: \"9c25987f-3a86-4c2e-ae7c-41080e0a3e24\") " Oct 30 19:25:21.285669 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:21.285489 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c25987f-3a86-4c2e-ae7c-41080e0a3e24-kube-api-access-7sbvz\" (UniqueName: \"kubernetes.io/projected/9c25987f-3a86-4c2e-ae7c-41080e0a3e24-kube-api-access-7sbvz\") pod \"9c25987f-3a86-4c2e-ae7c-41080e0a3e24\" (UID: \"9c25987f-3a86-4c2e-ae7c-41080e0a3e24\") " Oct 30 19:25:21.285669 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:21.285540 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c25987f-3a86-4c2e-ae7c-41080e0a3e24-cacerts\" (UniqueName: \"kubernetes.io/secret/9c25987f-3a86-4c2e-ae7c-41080e0a3e24-cacerts\") pod \"9c25987f-3a86-4c2e-ae7c-41080e0a3e24\" (UID: \"9c25987f-3a86-4c2e-ae7c-41080e0a3e24\") " Oct 30 19:25:21.285669 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:21.285623 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c25987f-3a86-4c2e-ae7c-41080e0a3e24-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9c25987f-3a86-4c2e-ae7c-41080e0a3e24-istio-kubeconfig\") pod \"9c25987f-3a86-4c2e-ae7c-41080e0a3e24\" (UID: \"9c25987f-3a86-4c2e-ae7c-41080e0a3e24\") " Oct 30 19:25:21.286217 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:21.285997 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c25987f-3a86-4c2e-ae7c-41080e0a3e24-istio-csr-ca-configmap" pod "9c25987f-3a86-4c2e-ae7c-41080e0a3e24" (UID: "9c25987f-3a86-4c2e-ae7c-41080e0a3e24"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:25:21.286217 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:21.286187 6960 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9c25987f-3a86-4c2e-ae7c-41080e0a3e24-istio-csr-ca-configmap\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:25:21.288244 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:21.288189 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c25987f-3a86-4c2e-ae7c-41080e0a3e24-local-certs" pod "9c25987f-3a86-4c2e-ae7c-41080e0a3e24" (UID: "9c25987f-3a86-4c2e-ae7c-41080e0a3e24"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:25:21.288458 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:21.288438 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c25987f-3a86-4c2e-ae7c-41080e0a3e24-istio-csr-dns-cert" pod "9c25987f-3a86-4c2e-ae7c-41080e0a3e24" (UID: "9c25987f-3a86-4c2e-ae7c-41080e0a3e24"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:25:21.288536 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:21.288456 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c25987f-3a86-4c2e-ae7c-41080e0a3e24-cacerts" pod "9c25987f-3a86-4c2e-ae7c-41080e0a3e24" (UID: "9c25987f-3a86-4c2e-ae7c-41080e0a3e24"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:25:21.288602 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:21.288532 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c25987f-3a86-4c2e-ae7c-41080e0a3e24-kube-api-access-7sbvz" pod "9c25987f-3a86-4c2e-ae7c-41080e0a3e24" (UID: "9c25987f-3a86-4c2e-ae7c-41080e0a3e24"). InnerVolumeSpecName "kube-api-access-7sbvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:25:21.288830 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:21.288810 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c25987f-3a86-4c2e-ae7c-41080e0a3e24-istio-kubeconfig" pod "9c25987f-3a86-4c2e-ae7c-41080e0a3e24" (UID: "9c25987f-3a86-4c2e-ae7c-41080e0a3e24"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:25:21.288873 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:21.288836 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c25987f-3a86-4c2e-ae7c-41080e0a3e24-istio-token" pod "9c25987f-3a86-4c2e-ae7c-41080e0a3e24" (UID: "9c25987f-3a86-4c2e-ae7c-41080e0a3e24"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:25:21.387198 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:21.387104 6960 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c25987f-3a86-4c2e-ae7c-41080e0a3e24-istio-token\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:25:21.387198 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:21.387131 6960 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9c25987f-3a86-4c2e-ae7c-41080e0a3e24-istio-csr-dns-cert\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:25:21.387198 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:21.387153 6960 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c25987f-3a86-4c2e-ae7c-41080e0a3e24-local-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:25:21.387198 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:21.387164 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7sbvz\" (UniqueName: \"kubernetes.io/projected/9c25987f-3a86-4c2e-ae7c-41080e0a3e24-kube-api-access-7sbvz\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:25:21.387198 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:21.387173 6960 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9c25987f-3a86-4c2e-ae7c-41080e0a3e24-cacerts\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:25:21.387198 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:21.387181 6960 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9c25987f-3a86-4c2e-ae7c-41080e0a3e24-istio-kubeconfig\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:25:21.909240 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:21.909210 6960 generic.go:356] "Generic (PLEG): container finished" podID="9c25987f-3a86-4c2e-ae7c-41080e0a3e24" containerID="ad3386ab527b9f935dbb792f3fb6567951172f5014c0e8b36b7a4b41629998ac" exitCode=0 Oct 30 19:25:21.909682 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:21.909254 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zgl9b" event={"ID":"9c25987f-3a86-4c2e-ae7c-41080e0a3e24","Type":"ContainerDied","Data":"ad3386ab527b9f935dbb792f3fb6567951172f5014c0e8b36b7a4b41629998ac"} Oct 30 19:25:21.909682 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:21.909258 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zgl9b" Oct 30 19:25:21.909682 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:21.909296 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zgl9b" event={"ID":"9c25987f-3a86-4c2e-ae7c-41080e0a3e24","Type":"ContainerDied","Data":"1a480c9c1637ea27124feda0c49f956d7389cc3a63cbf9e78df53c7183700581"} Oct 30 19:25:21.909682 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:21.909314 6960 scope.go:122] "RemoveContainer" containerID="ad3386ab527b9f935dbb792f3fb6567951172f5014c0e8b36b7a4b41629998ac" Oct 30 19:25:21.921898 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:21.921877 6960 scope.go:122] "RemoveContainer" containerID="ad3386ab527b9f935dbb792f3fb6567951172f5014c0e8b36b7a4b41629998ac" Oct 30 19:25:21.922216 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:25:21.922192 6960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad3386ab527b9f935dbb792f3fb6567951172f5014c0e8b36b7a4b41629998ac\": container with ID starting with ad3386ab527b9f935dbb792f3fb6567951172f5014c0e8b36b7a4b41629998ac not found: ID does not exist" containerID="ad3386ab527b9f935dbb792f3fb6567951172f5014c0e8b36b7a4b41629998ac" Oct 30 19:25:21.922280 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:21.922226 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad3386ab527b9f935dbb792f3fb6567951172f5014c0e8b36b7a4b41629998ac"} err="failed to get container status \"ad3386ab527b9f935dbb792f3fb6567951172f5014c0e8b36b7a4b41629998ac\": rpc error: code = NotFound desc = could not find container \"ad3386ab527b9f935dbb792f3fb6567951172f5014c0e8b36b7a4b41629998ac\": container with ID starting with ad3386ab527b9f935dbb792f3fb6567951172f5014c0e8b36b7a4b41629998ac not found: ID does not exist" Oct 30 19:25:21.936851 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:21.936828 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zgl9b"] Oct 30 19:25:21.941473 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:21.941450 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zgl9b"] Oct 30 19:25:22.139921 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:22.139883 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:22.592635 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:22.592604 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c25987f-3a86-4c2e-ae7c-41080e0a3e24" path="/var/lib/kubelet/pods/9c25987f-3a86-4c2e-ae7c-41080e0a3e24/volumes" Oct 30 19:25:23.145586 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:23.145549 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:23.931909 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:25:23.931872 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 19:25:24.244302 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:24.244210 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:25.173891 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:25.173844 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:25.360515 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:25.360475 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:26.472670 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:26.472630 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:27.305035 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:27.305000 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:28.131783 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:28.131744 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:28.995921 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:28.995869 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:29.795561 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:29.795529 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:30.651608 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:30.651567 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:31.427260 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:31.427233 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:32.320707 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:32.320669 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:33.215357 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:33.215323 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:34.004114 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:34.004080 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:34.892681 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:34.892635 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:35.560106 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:35.560073 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:35.679650 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:35.679590 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:35.790652 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:35.790613 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:36.493626 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:36.493588 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:37.179707 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:37.179669 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:37.406705 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:37.406674 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:37.560393 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:37.560353 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:38.058572 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:38.058531 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:38.509692 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:38.509589 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:39.312944 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:39.312914 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:40.103247 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:40.103214 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:41.011813 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:41.011770 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:41.893022 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:41.892980 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:42.724663 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:42.724627 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:43.743357 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:43.743325 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:44.630124 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:44.630074 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:45.682492 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:45.682452 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:45.810190 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:45.810127 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:46.063191 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:46.063154 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:46.670987 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:46.670951 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:47.913262 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:47.913217 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:48.630504 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:48.630467 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:49.409165 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:49.409102 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:50.216378 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:50.216342 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:51.116204 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:51.116170 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:52.000088 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:52.000055 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:52.808287 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:52.808250 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:53.717405 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:53.717353 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:53.931303 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:25:53.931268 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 19:25:54.601130 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:54.601094 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:54.888914 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:54.888814 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:55.406439 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:55.406407 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:56.192339 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:56.192310 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:56.772215 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:56.772182 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:56.892099 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:56.892050 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:57.105072 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:57.105032 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:57.974030 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:57.973995 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:58.671431 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:58.671391 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:25:59.504945 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:25:59.504908 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:00.188796 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:00.188757 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:00.372453 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:00.372413 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:00.485243 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:00.485131 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:00.592957 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:00.592926 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:01.025155 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:01.025102 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:01.413116 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:01.413080 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:02.287744 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:02.287710 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:03.136948 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:03.136909 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:04.154430 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:04.154397 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:05.056106 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:05.056061 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:05.915597 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:05.915564 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:06.716634 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:06.716602 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:07.810129 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:07.810099 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:08.612503 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:08.612469 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:09.111116 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:09.111079 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:09.273602 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:09.273572 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:09.519527 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:09.519444 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:09.676045 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:09.676010 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:10.826581 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:10.826549 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:11.721442 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:11.721408 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:12.687058 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:12.687024 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:13.539561 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:13.539530 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:14.337187 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:14.337125 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:15.115634 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:15.115600 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:15.924444 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:15.924409 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:17.230612 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:17.230579 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:17.705838 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:17.705795 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:18.618490 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:18.618454 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:19.397008 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:19.396974 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:20.296364 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:20.296328 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:20.415458 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:20.415426 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:21.210517 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:21.210485 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:22.027885 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:22.027850 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:22.813929 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:22.813895 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:23.836833 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:23.836794 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:23.932163 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:26:23.932110 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 19:26:24.572612 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:24.572570 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:24.774171 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:24.771525 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:24.908830 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:24.908742 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:25.034028 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:25.033996 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:25.390785 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:25.390744 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:25.907954 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:25.907918 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:26.850630 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:26.850597 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:27.608423 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:27.608387 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:28.452601 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:28.452570 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:29.291359 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:29.291263 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:30.112126 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:30.112097 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:30.897161 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:30.897109 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:31.687011 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:31.686973 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:32.571335 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:32.571305 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:32.712639 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:32.712603 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:32.989128 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:32.989033 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:33.124179 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:33.115457 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:33.696671 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:33.696580 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:34.721780 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:34.721741 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:35.598005 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:35.597971 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:36.531721 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:36.531691 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:37.314768 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:37.314736 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:38.208617 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:38.208581 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:39.206198 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:39.206128 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:40.033292 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:40.033255 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:40.802337 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:40.802297 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:41.628312 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:41.628274 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:42.599097 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:42.599061 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:43.416472 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:43.416436 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:43.774862 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:43.774789 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:43.885399 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:43.885368 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:44.133396 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:44.133362 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-f7c6e3-78d88cd6dd-pffm6"] Oct 30 19:26:44.133695 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:44.133668 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f7c6e3-78d88cd6dd-pffm6" podUID="aeba7fc6-3fe2-4327-94cb-84c86ad4f1d1" containerName="maas-api" containerID="cri-o://a2802603b8acf859798b9022e561737233e434dfc14081b24892c0c6f37a59d7" gracePeriod=30 Oct 30 19:26:44.185190 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:44.185155 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:44.396100 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:44.396015 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:44.406458 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:44.406424 6960 generic.go:356] "Generic (PLEG): container finished" podID="aeba7fc6-3fe2-4327-94cb-84c86ad4f1d1" containerID="a2802603b8acf859798b9022e561737233e434dfc14081b24892c0c6f37a59d7" exitCode=0 Oct 30 19:26:44.406689 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:44.406501 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f7c6e3-78d88cd6dd-pffm6" event={"ID":"aeba7fc6-3fe2-4327-94cb-84c86ad4f1d1","Type":"ContainerDied","Data":"a2802603b8acf859798b9022e561737233e434dfc14081b24892c0c6f37a59d7"} Oct 30 19:26:44.429278 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:44.429252 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f7c6e3-78d88cd6dd-pffm6" Oct 30 19:26:44.514961 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:44.514929 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aeba7fc6-3fe2-4327-94cb-84c86ad4f1d1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aeba7fc6-3fe2-4327-94cb-84c86ad4f1d1-openshift-service-ca\") pod \"aeba7fc6-3fe2-4327-94cb-84c86ad4f1d1\" (UID: \"aeba7fc6-3fe2-4327-94cb-84c86ad4f1d1\") " Oct 30 19:26:44.515203 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:44.514979 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aeba7fc6-3fe2-4327-94cb-84c86ad4f1d1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/aeba7fc6-3fe2-4327-94cb-84c86ad4f1d1-metrics-certs\") pod \"aeba7fc6-3fe2-4327-94cb-84c86ad4f1d1\" (UID: \"aeba7fc6-3fe2-4327-94cb-84c86ad4f1d1\") " Oct 30 19:26:44.515203 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:44.515016 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aeba7fc6-3fe2-4327-94cb-84c86ad4f1d1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aeba7fc6-3fe2-4327-94cb-84c86ad4f1d1-maas-api-tls\") pod \"aeba7fc6-3fe2-4327-94cb-84c86ad4f1d1\" (UID: \"aeba7fc6-3fe2-4327-94cb-84c86ad4f1d1\") " Oct 30 19:26:44.515203 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:44.515047 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aeba7fc6-3fe2-4327-94cb-84c86ad4f1d1-kube-api-access-sz7w6\" (UniqueName: \"kubernetes.io/projected/aeba7fc6-3fe2-4327-94cb-84c86ad4f1d1-kube-api-access-sz7w6\") pod \"aeba7fc6-3fe2-4327-94cb-84c86ad4f1d1\" (UID: \"aeba7fc6-3fe2-4327-94cb-84c86ad4f1d1\") " Oct 30 19:26:44.515438 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:44.515413 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aeba7fc6-3fe2-4327-94cb-84c86ad4f1d1-openshift-service-ca" pod "aeba7fc6-3fe2-4327-94cb-84c86ad4f1d1" (UID: "aeba7fc6-3fe2-4327-94cb-84c86ad4f1d1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:26:44.515655 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:44.515631 6960 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aeba7fc6-3fe2-4327-94cb-84c86ad4f1d1-openshift-service-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:26:44.517501 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:44.517471 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aeba7fc6-3fe2-4327-94cb-84c86ad4f1d1-kube-api-access-sz7w6" pod "aeba7fc6-3fe2-4327-94cb-84c86ad4f1d1" (UID: "aeba7fc6-3fe2-4327-94cb-84c86ad4f1d1"). InnerVolumeSpecName "kube-api-access-sz7w6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:26:44.517851 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:44.517828 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aeba7fc6-3fe2-4327-94cb-84c86ad4f1d1-maas-api-tls" pod "aeba7fc6-3fe2-4327-94cb-84c86ad4f1d1" (UID: "aeba7fc6-3fe2-4327-94cb-84c86ad4f1d1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:26:44.517928 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:44.517848 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aeba7fc6-3fe2-4327-94cb-84c86ad4f1d1-metrics-certs" pod "aeba7fc6-3fe2-4327-94cb-84c86ad4f1d1" (UID: "aeba7fc6-3fe2-4327-94cb-84c86ad4f1d1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:26:44.562411 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:44.562366 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:44.616443 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:44.616413 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sz7w6\" (UniqueName: \"kubernetes.io/projected/aeba7fc6-3fe2-4327-94cb-84c86ad4f1d1-kube-api-access-sz7w6\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:26:44.616443 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:44.616440 6960 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aeba7fc6-3fe2-4327-94cb-84c86ad4f1d1-metrics-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:26:44.616617 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:44.616454 6960 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aeba7fc6-3fe2-4327-94cb-84c86ad4f1d1-maas-api-tls\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:26:44.672390 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:44.672282 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:45.074047 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:45.074011 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:45.413715 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:45.413634 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f7c6e3-78d88cd6dd-pffm6" Oct 30 19:26:45.413715 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:45.413661 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f7c6e3-78d88cd6dd-pffm6" event={"ID":"aeba7fc6-3fe2-4327-94cb-84c86ad4f1d1","Type":"ContainerDied","Data":"3c5ba99be827caff4becabd5ef04dbb483bddade964ec158ad39ea319262cb29"} Oct 30 19:26:45.413968 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:45.413732 6960 scope.go:122] "RemoveContainer" containerID="a2802603b8acf859798b9022e561737233e434dfc14081b24892c0c6f37a59d7" Oct 30 19:26:45.431249 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:45.431216 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-f7c6e3-78d88cd6dd-pffm6"] Oct 30 19:26:45.435132 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:45.435111 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-f7c6e3-78d88cd6dd-pffm6"] Oct 30 19:26:45.508032 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:45.508003 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:46.311102 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:46.311071 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:46.592552 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:46.592474 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aeba7fc6-3fe2-4327-94cb-84c86ad4f1d1" path="/var/lib/kubelet/pods/aeba7fc6-3fe2-4327-94cb-84c86ad4f1d1/volumes" Oct 30 19:26:47.254821 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:47.254772 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:48.134669 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:48.134634 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:49.002433 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:49.002398 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:49.704024 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:49.703988 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:50.619012 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:50.618975 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:51.525029 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:51.524996 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:52.336831 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:52.336793 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:52.494277 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:52.494236 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:52.652079 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:52.652008 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-mswcj"] Oct 30 19:26:52.652571 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:52.652225 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-mswcj" podUID="f2fbe8d6-d7b0-4814-a665-e47be1a45176" containerName="istio-proxy" containerID="cri-o://0a85f9560a19dfd707d65ce74e658762b4f167dc607fd4ec82053e2debd1b265" gracePeriod=30 Oct 30 19:26:52.774003 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:52.773961 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:53.499050 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:53.499017 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:53.932926 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:26:53.932890 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 19:26:54.430086 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:54.430042 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:54.967983 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:54.967944 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:55.219672 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:55.219574 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:56.017506 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:56.017475 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:56.867096 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:56.866848 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:57.751887 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:57.751854 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:57.909095 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:57.909074 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-mswcj" Oct 30 19:26:58.076692 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:58.076662 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2fbe8d6-d7b0-4814-a665-e47be1a45176-istio-token\" (UniqueName: \"kubernetes.io/projected/f2fbe8d6-d7b0-4814-a665-e47be1a45176-istio-token\") pod \"f2fbe8d6-d7b0-4814-a665-e47be1a45176\" (UID: \"f2fbe8d6-d7b0-4814-a665-e47be1a45176\") " Oct 30 19:26:58.076886 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:58.076712 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2fbe8d6-d7b0-4814-a665-e47be1a45176-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2fbe8d6-d7b0-4814-a665-e47be1a45176-workload-certs\") pod \"f2fbe8d6-d7b0-4814-a665-e47be1a45176\" (UID: \"f2fbe8d6-d7b0-4814-a665-e47be1a45176\") " Oct 30 19:26:58.076886 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:58.076802 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f2fbe8d6-d7b0-4814-a665-e47be1a45176-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2fbe8d6-d7b0-4814-a665-e47be1a45176-istio-podinfo\") pod \"f2fbe8d6-d7b0-4814-a665-e47be1a45176\" (UID: \"f2fbe8d6-d7b0-4814-a665-e47be1a45176\") " Oct 30 19:26:58.076886 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:58.076829 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2fbe8d6-d7b0-4814-a665-e47be1a45176-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2fbe8d6-d7b0-4814-a665-e47be1a45176-credential-socket\") pod \"f2fbe8d6-d7b0-4814-a665-e47be1a45176\" (UID: \"f2fbe8d6-d7b0-4814-a665-e47be1a45176\") " Oct 30 19:26:58.076886 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:58.076866 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2fbe8d6-d7b0-4814-a665-e47be1a45176-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2fbe8d6-d7b0-4814-a665-e47be1a45176-istiod-ca-cert\") pod \"f2fbe8d6-d7b0-4814-a665-e47be1a45176\" (UID: \"f2fbe8d6-d7b0-4814-a665-e47be1a45176\") " Oct 30 19:26:58.077111 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:58.076897 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2fbe8d6-d7b0-4814-a665-e47be1a45176-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2fbe8d6-d7b0-4814-a665-e47be1a45176-istio-data\") pod \"f2fbe8d6-d7b0-4814-a665-e47be1a45176\" (UID: \"f2fbe8d6-d7b0-4814-a665-e47be1a45176\") " Oct 30 19:26:58.077111 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:58.076943 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2fbe8d6-d7b0-4814-a665-e47be1a45176-kube-api-access-n79rg\" (UniqueName: \"kubernetes.io/projected/f2fbe8d6-d7b0-4814-a665-e47be1a45176-kube-api-access-n79rg\") pod \"f2fbe8d6-d7b0-4814-a665-e47be1a45176\" (UID: \"f2fbe8d6-d7b0-4814-a665-e47be1a45176\") " Oct 30 19:26:58.077111 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:58.077014 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2fbe8d6-d7b0-4814-a665-e47be1a45176-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2fbe8d6-d7b0-4814-a665-e47be1a45176-workload-socket\") pod \"f2fbe8d6-d7b0-4814-a665-e47be1a45176\" (UID: \"f2fbe8d6-d7b0-4814-a665-e47be1a45176\") " Oct 30 19:26:58.077111 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:58.077061 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2fbe8d6-d7b0-4814-a665-e47be1a45176-workload-certs" pod "f2fbe8d6-d7b0-4814-a665-e47be1a45176" (UID: "f2fbe8d6-d7b0-4814-a665-e47be1a45176"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:26:58.077111 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:58.077089 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2fbe8d6-d7b0-4814-a665-e47be1a45176-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2fbe8d6-d7b0-4814-a665-e47be1a45176-istio-envoy\") pod \"f2fbe8d6-d7b0-4814-a665-e47be1a45176\" (UID: \"f2fbe8d6-d7b0-4814-a665-e47be1a45176\") " Oct 30 19:26:58.077413 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:58.077245 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2fbe8d6-d7b0-4814-a665-e47be1a45176-credential-socket" pod "f2fbe8d6-d7b0-4814-a665-e47be1a45176" (UID: "f2fbe8d6-d7b0-4814-a665-e47be1a45176"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:26:58.077413 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:58.077261 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2fbe8d6-d7b0-4814-a665-e47be1a45176-istio-data" pod "f2fbe8d6-d7b0-4814-a665-e47be1a45176" (UID: "f2fbe8d6-d7b0-4814-a665-e47be1a45176"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:26:58.077413 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:58.077333 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2fbe8d6-d7b0-4814-a665-e47be1a45176-workload-socket" pod "f2fbe8d6-d7b0-4814-a665-e47be1a45176" (UID: "f2fbe8d6-d7b0-4814-a665-e47be1a45176"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:26:58.077413 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:58.077340 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2fbe8d6-d7b0-4814-a665-e47be1a45176-istiod-ca-cert" pod "f2fbe8d6-d7b0-4814-a665-e47be1a45176" (UID: "f2fbe8d6-d7b0-4814-a665-e47be1a45176"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:26:58.077708 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:58.077573 6960 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2fbe8d6-d7b0-4814-a665-e47be1a45176-credential-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:26:58.077708 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:58.077589 6960 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2fbe8d6-d7b0-4814-a665-e47be1a45176-istiod-ca-cert\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:26:58.077708 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:58.077599 6960 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2fbe8d6-d7b0-4814-a665-e47be1a45176-istio-data\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:26:58.077708 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:58.077608 6960 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2fbe8d6-d7b0-4814-a665-e47be1a45176-workload-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:26:58.077708 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:58.077620 6960 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2fbe8d6-d7b0-4814-a665-e47be1a45176-workload-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:26:58.079253 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:58.079227 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f2fbe8d6-d7b0-4814-a665-e47be1a45176-istio-podinfo" pod "f2fbe8d6-d7b0-4814-a665-e47be1a45176" (UID: "f2fbe8d6-d7b0-4814-a665-e47be1a45176"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:26:58.079521 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:58.079499 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2fbe8d6-d7b0-4814-a665-e47be1a45176-istio-envoy" pod "f2fbe8d6-d7b0-4814-a665-e47be1a45176" (UID: "f2fbe8d6-d7b0-4814-a665-e47be1a45176"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:26:58.079688 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:58.079665 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2fbe8d6-d7b0-4814-a665-e47be1a45176-istio-token" pod "f2fbe8d6-d7b0-4814-a665-e47be1a45176" (UID: "f2fbe8d6-d7b0-4814-a665-e47be1a45176"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:26:58.079688 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:58.079679 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2fbe8d6-d7b0-4814-a665-e47be1a45176-kube-api-access-n79rg" pod "f2fbe8d6-d7b0-4814-a665-e47be1a45176" (UID: "f2fbe8d6-d7b0-4814-a665-e47be1a45176"). InnerVolumeSpecName "kube-api-access-n79rg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:26:58.179138 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:58.179112 6960 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2fbe8d6-d7b0-4814-a665-e47be1a45176-istio-token\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:26:58.179138 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:58.179135 6960 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2fbe8d6-d7b0-4814-a665-e47be1a45176-istio-podinfo\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:26:58.179309 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:58.179174 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n79rg\" (UniqueName: \"kubernetes.io/projected/f2fbe8d6-d7b0-4814-a665-e47be1a45176-kube-api-access-n79rg\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:26:58.179309 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:58.179183 6960 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2fbe8d6-d7b0-4814-a665-e47be1a45176-istio-envoy\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:26:58.485455 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:58.485370 6960 generic.go:356] "Generic (PLEG): container finished" podID="f2fbe8d6-d7b0-4814-a665-e47be1a45176" containerID="0a85f9560a19dfd707d65ce74e658762b4f167dc607fd4ec82053e2debd1b265" exitCode=0 Oct 30 19:26:58.485455 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:58.485420 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-mswcj" Oct 30 19:26:58.485657 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:58.485452 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-mswcj" event={"ID":"f2fbe8d6-d7b0-4814-a665-e47be1a45176","Type":"ContainerDied","Data":"0a85f9560a19dfd707d65ce74e658762b4f167dc607fd4ec82053e2debd1b265"} Oct 30 19:26:58.485657 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:58.485513 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-mswcj" event={"ID":"f2fbe8d6-d7b0-4814-a665-e47be1a45176","Type":"ContainerDied","Data":"7b305c48c5f27da40aaaceda012848a083f35fee0b75526ac9e5c1d973a3232d"} Oct 30 19:26:58.485657 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:58.485543 6960 scope.go:122] "RemoveContainer" containerID="0a85f9560a19dfd707d65ce74e658762b4f167dc607fd4ec82053e2debd1b265" Oct 30 19:26:58.496649 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:58.496626 6960 scope.go:122] "RemoveContainer" containerID="0a85f9560a19dfd707d65ce74e658762b4f167dc607fd4ec82053e2debd1b265" Oct 30 19:26:58.496916 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:26:58.496894 6960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a85f9560a19dfd707d65ce74e658762b4f167dc607fd4ec82053e2debd1b265\": container with ID starting with 0a85f9560a19dfd707d65ce74e658762b4f167dc607fd4ec82053e2debd1b265 not found: ID does not exist" containerID="0a85f9560a19dfd707d65ce74e658762b4f167dc607fd4ec82053e2debd1b265" Oct 30 19:26:58.497007 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:58.496924 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a85f9560a19dfd707d65ce74e658762b4f167dc607fd4ec82053e2debd1b265"} err="failed to get container status \"0a85f9560a19dfd707d65ce74e658762b4f167dc607fd4ec82053e2debd1b265\": rpc error: code = NotFound desc = could not find container \"0a85f9560a19dfd707d65ce74e658762b4f167dc607fd4ec82053e2debd1b265\": container with ID starting with 0a85f9560a19dfd707d65ce74e658762b4f167dc607fd4ec82053e2debd1b265 not found: ID does not exist" Oct 30 19:26:58.508675 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:58.508651 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-mswcj"] Oct 30 19:26:58.511710 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:58.511686 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-f7c6e3-openshift-default-68f95fdc7-mswcj"] Oct 30 19:26:58.592192 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:58.592129 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2fbe8d6-d7b0-4814-a665-e47be1a45176" path="/var/lib/kubelet/pods/f2fbe8d6-d7b0-4814-a665-e47be1a45176/volumes" Oct 30 19:26:58.625399 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:58.625356 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:26:59.404913 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:26:59.404880 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:00.212707 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:00.212671 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:01.126833 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:01.126797 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:02.020784 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:02.020733 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:02.906358 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:02.906317 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:03.279673 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:03.279598 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-e1e1ff-848c46dccb-rbs5r"] Oct 30 19:27:03.305656 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:03.305626 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e1e1ff-5f47bcf47f-2wj2p"] Oct 30 19:27:03.369990 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:03.369957 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:03.506674 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:03.506639 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:03.717259 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:03.717220 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:04.535565 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:04.535530 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:05.363386 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:05.363348 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:05.961193 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:05.961161 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e1e1ff-5d6f9c8b5-xsp5m"] Oct 30 19:27:05.961949 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:05.961614 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e1e1ff-5d6f9c8b5-xsp5m" podUID="48c4bb82-3969-4e2d-a303-ef614827b953" containerName="maas-api" containerID="cri-o://fd345babda3c5c2a832a518ff73349085977ec2afb294ad7baa56754aabab6eb" gracePeriod=30 Oct 30 19:27:06.017992 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:06.017955 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:06.254003 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:06.253979 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e1e1ff-5d6f9c8b5-xsp5m" Oct 30 19:27:06.259601 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:06.259565 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:06.267831 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:06.267809 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48c4bb82-3969-4e2d-a303-ef614827b953-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48c4bb82-3969-4e2d-a303-ef614827b953-maas-api-tls\") pod \"48c4bb82-3969-4e2d-a303-ef614827b953\" (UID: \"48c4bb82-3969-4e2d-a303-ef614827b953\") " Oct 30 19:27:06.267974 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:06.267874 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48c4bb82-3969-4e2d-a303-ef614827b953-kube-api-access-r7wrw\" (UniqueName: \"kubernetes.io/projected/48c4bb82-3969-4e2d-a303-ef614827b953-kube-api-access-r7wrw\") pod \"48c4bb82-3969-4e2d-a303-ef614827b953\" (UID: \"48c4bb82-3969-4e2d-a303-ef614827b953\") " Oct 30 19:27:06.267974 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:06.267966 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48c4bb82-3969-4e2d-a303-ef614827b953-metrics-certs\" (UniqueName: \"kubernetes.io/secret/48c4bb82-3969-4e2d-a303-ef614827b953-metrics-certs\") pod \"48c4bb82-3969-4e2d-a303-ef614827b953\" (UID: \"48c4bb82-3969-4e2d-a303-ef614827b953\") " Oct 30 19:27:06.268101 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:06.268005 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48c4bb82-3969-4e2d-a303-ef614827b953-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48c4bb82-3969-4e2d-a303-ef614827b953-openshift-service-ca\") pod \"48c4bb82-3969-4e2d-a303-ef614827b953\" (UID: \"48c4bb82-3969-4e2d-a303-ef614827b953\") " Oct 30 19:27:06.268425 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:06.268403 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48c4bb82-3969-4e2d-a303-ef614827b953-openshift-service-ca" pod "48c4bb82-3969-4e2d-a303-ef614827b953" (UID: "48c4bb82-3969-4e2d-a303-ef614827b953"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:27:06.270308 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:06.270278 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48c4bb82-3969-4e2d-a303-ef614827b953-kube-api-access-r7wrw" pod "48c4bb82-3969-4e2d-a303-ef614827b953" (UID: "48c4bb82-3969-4e2d-a303-ef614827b953"). InnerVolumeSpecName "kube-api-access-r7wrw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:27:06.270308 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:06.270286 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48c4bb82-3969-4e2d-a303-ef614827b953-metrics-certs" pod "48c4bb82-3969-4e2d-a303-ef614827b953" (UID: "48c4bb82-3969-4e2d-a303-ef614827b953"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:27:06.270460 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:06.270396 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48c4bb82-3969-4e2d-a303-ef614827b953-maas-api-tls" pod "48c4bb82-3969-4e2d-a303-ef614827b953" (UID: "48c4bb82-3969-4e2d-a303-ef614827b953"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:27:06.369799 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:06.369770 6960 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48c4bb82-3969-4e2d-a303-ef614827b953-metrics-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:27:06.369799 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:06.369799 6960 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48c4bb82-3969-4e2d-a303-ef614827b953-openshift-service-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:27:06.369799 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:06.369809 6960 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48c4bb82-3969-4e2d-a303-ef614827b953-maas-api-tls\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:27:06.370045 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:06.369822 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7wrw\" (UniqueName: \"kubernetes.io/projected/48c4bb82-3969-4e2d-a303-ef614827b953-kube-api-access-r7wrw\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:27:06.412014 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:06.411976 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:06.521546 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:06.521450 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:06.532188 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:06.532155 6960 generic.go:356] "Generic (PLEG): container finished" podID="48c4bb82-3969-4e2d-a303-ef614827b953" containerID="fd345babda3c5c2a832a518ff73349085977ec2afb294ad7baa56754aabab6eb" exitCode=0 Oct 30 19:27:06.532360 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:06.532197 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e1e1ff-5d6f9c8b5-xsp5m" event={"ID":"48c4bb82-3969-4e2d-a303-ef614827b953","Type":"ContainerDied","Data":"fd345babda3c5c2a832a518ff73349085977ec2afb294ad7baa56754aabab6eb"} Oct 30 19:27:06.532360 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:06.532235 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e1e1ff-5d6f9c8b5-xsp5m" Oct 30 19:27:06.532360 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:06.532304 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e1e1ff-5d6f9c8b5-xsp5m" event={"ID":"48c4bb82-3969-4e2d-a303-ef614827b953","Type":"ContainerDied","Data":"8ae0e2b7ed57d6e0c706e9a14e06c386ee51651e9c227b3595c4709fbc1a45af"} Oct 30 19:27:06.532360 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:06.532340 6960 scope.go:122] "RemoveContainer" containerID="fd345babda3c5c2a832a518ff73349085977ec2afb294ad7baa56754aabab6eb" Oct 30 19:27:06.542478 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:06.542461 6960 scope.go:122] "RemoveContainer" containerID="fd345babda3c5c2a832a518ff73349085977ec2afb294ad7baa56754aabab6eb" Oct 30 19:27:06.542764 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:27:06.542740 6960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd345babda3c5c2a832a518ff73349085977ec2afb294ad7baa56754aabab6eb\": container with ID starting with fd345babda3c5c2a832a518ff73349085977ec2afb294ad7baa56754aabab6eb not found: ID does not exist" containerID="fd345babda3c5c2a832a518ff73349085977ec2afb294ad7baa56754aabab6eb" Oct 30 19:27:06.542829 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:06.542777 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd345babda3c5c2a832a518ff73349085977ec2afb294ad7baa56754aabab6eb"} err="failed to get container status \"fd345babda3c5c2a832a518ff73349085977ec2afb294ad7baa56754aabab6eb\": rpc error: code = NotFound desc = could not find container \"fd345babda3c5c2a832a518ff73349085977ec2afb294ad7baa56754aabab6eb\": container with ID starting with fd345babda3c5c2a832a518ff73349085977ec2afb294ad7baa56754aabab6eb not found: ID does not exist" Oct 30 19:27:06.556420 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:06.556390 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e1e1ff-5d6f9c8b5-xsp5m"] Oct 30 19:27:06.563233 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:06.563213 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-e1e1ff-5d6f9c8b5-xsp5m"] Oct 30 19:27:06.593706 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:06.593677 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48c4bb82-3969-4e2d-a303-ef614827b953" path="/var/lib/kubelet/pods/48c4bb82-3969-4e2d-a303-ef614827b953/volumes" Oct 30 19:27:06.895810 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:06.895771 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:07.326127 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:07.326093 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:08.280203 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:08.280160 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e1e1ff-848c46dccb-rbs5r" podUID="6804806b-3a5b-425d-9ae7-55bda5f68a4e" containerName="payload-processing" containerID="cri-o://4b8015039308c712a31052f4da61fc06559002366e60f307cde54bf586651785" gracePeriod=25 Oct 30 19:27:08.306974 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:08.306938 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e1e1ff-5f47bcf47f-2wj2p" podUID="ccb27153-3ebf-4eaa-80a1-e460de5d78d0" containerName="payload-pre-processing" containerID="cri-o://0c6d78f830007a990d4be79959873f72c8a088d132f06576bc19821e8c5bf572" gracePeriod=25 Oct 30 19:27:08.411111 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:08.411066 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:08.549206 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:08.549175 6960 generic.go:356] "Generic (PLEG): container finished" podID="ccb27153-3ebf-4eaa-80a1-e460de5d78d0" containerID="0c6d78f830007a990d4be79959873f72c8a088d132f06576bc19821e8c5bf572" exitCode=0 Oct 30 19:27:08.549379 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:08.549356 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e1e1ff-5f47bcf47f-2wj2p" event={"ID":"ccb27153-3ebf-4eaa-80a1-e460de5d78d0","Type":"ContainerDied","Data":"0c6d78f830007a990d4be79959873f72c8a088d132f06576bc19821e8c5bf572"} Oct 30 19:27:08.551098 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:08.551074 6960 generic.go:356] "Generic (PLEG): container finished" podID="6804806b-3a5b-425d-9ae7-55bda5f68a4e" containerID="4b8015039308c712a31052f4da61fc06559002366e60f307cde54bf586651785" exitCode=0 Oct 30 19:27:08.551235 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:08.551134 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e1e1ff-848c46dccb-rbs5r" event={"ID":"6804806b-3a5b-425d-9ae7-55bda5f68a4e","Type":"ContainerDied","Data":"4b8015039308c712a31052f4da61fc06559002366e60f307cde54bf586651785"} Oct 30 19:27:08.568226 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:08.568202 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e1e1ff-848c46dccb-rbs5r" Oct 30 19:27:08.593462 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:08.593433 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6804806b-3a5b-425d-9ae7-55bda5f68a4e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6804806b-3a5b-425d-9ae7-55bda5f68a4e-plugins-config-volume\") pod \"6804806b-3a5b-425d-9ae7-55bda5f68a4e\" (UID: \"6804806b-3a5b-425d-9ae7-55bda5f68a4e\") " Oct 30 19:27:08.593590 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:08.593503 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6804806b-3a5b-425d-9ae7-55bda5f68a4e-kube-api-access-nbc9b\" (UniqueName: \"kubernetes.io/projected/6804806b-3a5b-425d-9ae7-55bda5f68a4e-kube-api-access-nbc9b\") pod \"6804806b-3a5b-425d-9ae7-55bda5f68a4e\" (UID: \"6804806b-3a5b-425d-9ae7-55bda5f68a4e\") " Oct 30 19:27:08.593865 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:08.593826 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6804806b-3a5b-425d-9ae7-55bda5f68a4e-plugins-config-volume" pod "6804806b-3a5b-425d-9ae7-55bda5f68a4e" (UID: "6804806b-3a5b-425d-9ae7-55bda5f68a4e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:27:08.594048 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:08.594014 6960 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6804806b-3a5b-425d-9ae7-55bda5f68a4e-plugins-config-volume\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:27:08.596290 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:08.596261 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6804806b-3a5b-425d-9ae7-55bda5f68a4e-kube-api-access-nbc9b" pod "6804806b-3a5b-425d-9ae7-55bda5f68a4e" (UID: "6804806b-3a5b-425d-9ae7-55bda5f68a4e"). InnerVolumeSpecName "kube-api-access-nbc9b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:27:08.600618 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:08.600601 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e1e1ff-5f47bcf47f-2wj2p" Oct 30 19:27:08.694420 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:08.694389 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccb27153-3ebf-4eaa-80a1-e460de5d78d0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccb27153-3ebf-4eaa-80a1-e460de5d78d0-plugins-config-volume\") pod \"ccb27153-3ebf-4eaa-80a1-e460de5d78d0\" (UID: \"ccb27153-3ebf-4eaa-80a1-e460de5d78d0\") " Oct 30 19:27:08.694640 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:08.694488 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccb27153-3ebf-4eaa-80a1-e460de5d78d0-kube-api-access-tjb99\" (UniqueName: \"kubernetes.io/projected/ccb27153-3ebf-4eaa-80a1-e460de5d78d0-kube-api-access-tjb99\") pod \"ccb27153-3ebf-4eaa-80a1-e460de5d78d0\" (UID: \"ccb27153-3ebf-4eaa-80a1-e460de5d78d0\") " Oct 30 19:27:08.694748 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:08.694726 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccb27153-3ebf-4eaa-80a1-e460de5d78d0-plugins-config-volume" pod "ccb27153-3ebf-4eaa-80a1-e460de5d78d0" (UID: "ccb27153-3ebf-4eaa-80a1-e460de5d78d0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:27:08.694927 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:08.694905 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbc9b\" (UniqueName: \"kubernetes.io/projected/6804806b-3a5b-425d-9ae7-55bda5f68a4e-kube-api-access-nbc9b\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:27:08.694994 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:08.694934 6960 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccb27153-3ebf-4eaa-80a1-e460de5d78d0-plugins-config-volume\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:27:08.696821 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:08.696802 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccb27153-3ebf-4eaa-80a1-e460de5d78d0-kube-api-access-tjb99" pod "ccb27153-3ebf-4eaa-80a1-e460de5d78d0" (UID: "ccb27153-3ebf-4eaa-80a1-e460de5d78d0"). InnerVolumeSpecName "kube-api-access-tjb99". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:27:08.796285 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:08.796207 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjb99\" (UniqueName: \"kubernetes.io/projected/ccb27153-3ebf-4eaa-80a1-e460de5d78d0-kube-api-access-tjb99\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:27:09.312208 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:09.312164 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:09.561584 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:09.561558 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e1e1ff-5f47bcf47f-2wj2p" Oct 30 19:27:09.561584 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:09.561566 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e1e1ff-5f47bcf47f-2wj2p" event={"ID":"ccb27153-3ebf-4eaa-80a1-e460de5d78d0","Type":"ContainerDied","Data":"ba951fb17792ed3ce4349d99d77bb240fb290522c42911bd9256c15a96166e1e"} Oct 30 19:27:09.562171 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:09.561643 6960 scope.go:122] "RemoveContainer" containerID="0c6d78f830007a990d4be79959873f72c8a088d132f06576bc19821e8c5bf572" Oct 30 19:27:09.563570 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:09.563543 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e1e1ff-848c46dccb-rbs5r" event={"ID":"6804806b-3a5b-425d-9ae7-55bda5f68a4e","Type":"ContainerDied","Data":"145754ca7a44a4a136c04407d8b850a2d2c78d1dcdea8432902d9f081050a75d"} Oct 30 19:27:09.563686 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:09.563559 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e1e1ff-848c46dccb-rbs5r" Oct 30 19:27:09.574668 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:09.574647 6960 scope.go:122] "RemoveContainer" containerID="4b8015039308c712a31052f4da61fc06559002366e60f307cde54bf586651785" Oct 30 19:27:09.593218 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:09.593194 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-e1e1ff-848c46dccb-rbs5r"] Oct 30 19:27:09.595914 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:09.595892 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-e1e1ff-848c46dccb-rbs5r"] Oct 30 19:27:09.607775 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:09.607750 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e1e1ff-5f47bcf47f-2wj2p"] Oct 30 19:27:09.610108 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:09.610089 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e1e1ff-5f47bcf47f-2wj2p"] Oct 30 19:27:10.285548 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:10.285499 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:10.592056 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:10.592024 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6804806b-3a5b-425d-9ae7-55bda5f68a4e" path="/var/lib/kubelet/pods/6804806b-3a5b-425d-9ae7-55bda5f68a4e/volumes" Oct 30 19:27:10.592472 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:10.592449 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccb27153-3ebf-4eaa-80a1-e460de5d78d0" path="/var/lib/kubelet/pods/ccb27153-3ebf-4eaa-80a1-e460de5d78d0/volumes" Oct 30 19:27:11.060643 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:11.060583 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:12.075639 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:12.075602 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:12.914946 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:12.914909 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:13.816464 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:13.816422 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:14.086696 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:14.086600 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:14.196351 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:14.196317 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:14.450928 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:14.450838 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:14.981342 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:14.981301 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:15.999010 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:15.998972 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:16.825604 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:16.825570 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:17.734315 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:17.734279 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:18.610054 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:18.610020 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:19.500178 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:19.500124 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:20.307315 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:20.307278 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:21.111377 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:21.111342 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:22.109952 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:22.109917 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:22.910695 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:22.910663 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:23.712207 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:23.712172 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:23.931694 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:27:23.931661 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 19:27:24.604238 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:24.604212 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:25.109272 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:25.109236 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-b0e009-bfd774574-dl2n4"] Oct 30 19:27:25.120713 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:25.120677 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b0e009-bb65989f9-m7q2f"] Oct 30 19:27:25.194275 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:25.194238 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:25.311820 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:25.311781 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:25.451924 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:25.451801 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:26.197105 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:26.197074 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:26.825443 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:26.825398 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b0e009-5c7656bc96-lqqdw"] Oct 30 19:27:26.825673 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:26.825649 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b0e009-5c7656bc96-lqqdw" podUID="696a841d-b511-4430-a236-d61ff1832626" containerName="maas-api" containerID="cri-o://976cdf277796caa6e7cc51d599a4e19494c24270be5d1306546bf69c4a967112" gracePeriod=30 Oct 30 19:27:26.898708 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:26.898668 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:27.095207 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:27.095080 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:27.116532 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:27.116512 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b0e009-5c7656bc96-lqqdw" Oct 30 19:27:27.272796 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:27.272761 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:27.293859 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:27.293831 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/696a841d-b511-4430-a236-d61ff1832626-metrics-certs\" (UniqueName: \"kubernetes.io/secret/696a841d-b511-4430-a236-d61ff1832626-metrics-certs\") pod \"696a841d-b511-4430-a236-d61ff1832626\" (UID: \"696a841d-b511-4430-a236-d61ff1832626\") " Oct 30 19:27:27.294011 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:27.293887 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/696a841d-b511-4430-a236-d61ff1832626-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/696a841d-b511-4430-a236-d61ff1832626-maas-api-tls\") pod \"696a841d-b511-4430-a236-d61ff1832626\" (UID: \"696a841d-b511-4430-a236-d61ff1832626\") " Oct 30 19:27:27.294011 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:27.293914 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/696a841d-b511-4430-a236-d61ff1832626-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/696a841d-b511-4430-a236-d61ff1832626-openshift-service-ca\") pod \"696a841d-b511-4430-a236-d61ff1832626\" (UID: \"696a841d-b511-4430-a236-d61ff1832626\") " Oct 30 19:27:27.294011 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:27.293970 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/696a841d-b511-4430-a236-d61ff1832626-kube-api-access-snhn5\" (UniqueName: \"kubernetes.io/projected/696a841d-b511-4430-a236-d61ff1832626-kube-api-access-snhn5\") pod \"696a841d-b511-4430-a236-d61ff1832626\" (UID: \"696a841d-b511-4430-a236-d61ff1832626\") " Oct 30 19:27:27.294386 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:27.294359 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/696a841d-b511-4430-a236-d61ff1832626-openshift-service-ca" pod "696a841d-b511-4430-a236-d61ff1832626" (UID: "696a841d-b511-4430-a236-d61ff1832626"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:27:27.296194 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:27.296165 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/696a841d-b511-4430-a236-d61ff1832626-metrics-certs" pod "696a841d-b511-4430-a236-d61ff1832626" (UID: "696a841d-b511-4430-a236-d61ff1832626"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:27:27.296311 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:27.296299 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/696a841d-b511-4430-a236-d61ff1832626-kube-api-access-snhn5" pod "696a841d-b511-4430-a236-d61ff1832626" (UID: "696a841d-b511-4430-a236-d61ff1832626"). InnerVolumeSpecName "kube-api-access-snhn5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:27:27.296378 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:27.296338 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/696a841d-b511-4430-a236-d61ff1832626-maas-api-tls" pod "696a841d-b511-4430-a236-d61ff1832626" (UID: "696a841d-b511-4430-a236-d61ff1832626"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:27:27.395168 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:27.395066 6960 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/696a841d-b511-4430-a236-d61ff1832626-metrics-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:27:27.395168 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:27.395094 6960 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/696a841d-b511-4430-a236-d61ff1832626-maas-api-tls\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:27:27.395168 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:27.395107 6960 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/696a841d-b511-4430-a236-d61ff1832626-openshift-service-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:27:27.395168 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:27.395116 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snhn5\" (UniqueName: \"kubernetes.io/projected/696a841d-b511-4430-a236-d61ff1832626-kube-api-access-snhn5\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:27:27.405093 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:27.405062 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:27.667375 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:27.667285 6960 generic.go:356] "Generic (PLEG): container finished" podID="696a841d-b511-4430-a236-d61ff1832626" containerID="976cdf277796caa6e7cc51d599a4e19494c24270be5d1306546bf69c4a967112" exitCode=0 Oct 30 19:27:27.667375 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:27.667326 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b0e009-5c7656bc96-lqqdw" Oct 30 19:27:27.667375 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:27.667353 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b0e009-5c7656bc96-lqqdw" event={"ID":"696a841d-b511-4430-a236-d61ff1832626","Type":"ContainerDied","Data":"976cdf277796caa6e7cc51d599a4e19494c24270be5d1306546bf69c4a967112"} Oct 30 19:27:27.667668 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:27.667406 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b0e009-5c7656bc96-lqqdw" event={"ID":"696a841d-b511-4430-a236-d61ff1832626","Type":"ContainerDied","Data":"1390c906816f8a943a45eb7f5e00700489591831b56732f12acc6d200333a387"} Oct 30 19:27:27.667668 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:27.667433 6960 scope.go:122] "RemoveContainer" containerID="976cdf277796caa6e7cc51d599a4e19494c24270be5d1306546bf69c4a967112" Oct 30 19:27:27.677568 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:27.677551 6960 scope.go:122] "RemoveContainer" containerID="976cdf277796caa6e7cc51d599a4e19494c24270be5d1306546bf69c4a967112" Oct 30 19:27:27.677825 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:27:27.677806 6960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"976cdf277796caa6e7cc51d599a4e19494c24270be5d1306546bf69c4a967112\": container with ID starting with 976cdf277796caa6e7cc51d599a4e19494c24270be5d1306546bf69c4a967112 not found: ID does not exist" containerID="976cdf277796caa6e7cc51d599a4e19494c24270be5d1306546bf69c4a967112" Oct 30 19:27:27.677876 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:27.677834 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"976cdf277796caa6e7cc51d599a4e19494c24270be5d1306546bf69c4a967112"} err="failed to get container status \"976cdf277796caa6e7cc51d599a4e19494c24270be5d1306546bf69c4a967112\": rpc error: code = NotFound desc = could not find container \"976cdf277796caa6e7cc51d599a4e19494c24270be5d1306546bf69c4a967112\": container with ID starting with 976cdf277796caa6e7cc51d599a4e19494c24270be5d1306546bf69c4a967112 not found: ID does not exist" Oct 30 19:27:27.688950 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:27.688928 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b0e009-5c7656bc96-lqqdw"] Oct 30 19:27:27.693258 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:27.693238 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b0e009-5c7656bc96-lqqdw"] Oct 30 19:27:27.822765 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:27.822726 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:28.122184 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:28.122127 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:28.591926 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:28.591898 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="696a841d-b511-4430-a236-d61ff1832626" path="/var/lib/kubelet/pods/696a841d-b511-4430-a236-d61ff1832626/volumes" Oct 30 19:27:29.123291 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:29.123248 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:29.960980 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:29.960940 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:30.109690 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:30.109649 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b0e009-bfd774574-dl2n4" podUID="f3541c21-2190-4ba9-a2aa-286582db6f34" containerName="payload-processing" containerID="cri-o://18b412ec06a44da429e002741338dc60551bb66300e93211144a06ae621d4680" gracePeriod=25 Oct 30 19:27:30.121226 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:30.121190 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b0e009-bb65989f9-m7q2f" podUID="6ebea224-1a00-496f-a41e-1d7f2a294a5b" containerName="payload-pre-processing" containerID="cri-o://797f04cb0740af078454d1f763dbfcf7ebf5c88bd63029ccdd5bf0e4fbe1dba7" gracePeriod=25 Oct 30 19:27:30.371124 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:30.371098 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b0e009-bfd774574-dl2n4" Oct 30 19:27:30.399072 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:30.399049 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b0e009-bb65989f9-m7q2f" Oct 30 19:27:30.534879 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:30.534799 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ebea224-1a00-496f-a41e-1d7f2a294a5b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ebea224-1a00-496f-a41e-1d7f2a294a5b-plugins-config-volume\") pod \"6ebea224-1a00-496f-a41e-1d7f2a294a5b\" (UID: \"6ebea224-1a00-496f-a41e-1d7f2a294a5b\") " Oct 30 19:27:30.535030 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:30.534894 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3541c21-2190-4ba9-a2aa-286582db6f34-kube-api-access-8fgps\" (UniqueName: \"kubernetes.io/projected/f3541c21-2190-4ba9-a2aa-286582db6f34-kube-api-access-8fgps\") pod \"f3541c21-2190-4ba9-a2aa-286582db6f34\" (UID: \"f3541c21-2190-4ba9-a2aa-286582db6f34\") " Oct 30 19:27:30.535030 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:30.534985 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3541c21-2190-4ba9-a2aa-286582db6f34-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3541c21-2190-4ba9-a2aa-286582db6f34-plugins-config-volume\") pod \"f3541c21-2190-4ba9-a2aa-286582db6f34\" (UID: \"f3541c21-2190-4ba9-a2aa-286582db6f34\") " Oct 30 19:27:30.535185 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:30.535056 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ebea224-1a00-496f-a41e-1d7f2a294a5b-kube-api-access-nmmgh\" (UniqueName: \"kubernetes.io/projected/6ebea224-1a00-496f-a41e-1d7f2a294a5b-kube-api-access-nmmgh\") pod \"6ebea224-1a00-496f-a41e-1d7f2a294a5b\" (UID: \"6ebea224-1a00-496f-a41e-1d7f2a294a5b\") " Oct 30 19:27:30.535185 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:30.535170 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ebea224-1a00-496f-a41e-1d7f2a294a5b-plugins-config-volume" pod "6ebea224-1a00-496f-a41e-1d7f2a294a5b" (UID: "6ebea224-1a00-496f-a41e-1d7f2a294a5b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:27:30.535427 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:30.535405 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3541c21-2190-4ba9-a2aa-286582db6f34-plugins-config-volume" pod "f3541c21-2190-4ba9-a2aa-286582db6f34" (UID: "f3541c21-2190-4ba9-a2aa-286582db6f34"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:27:30.535537 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:30.535521 6960 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ebea224-1a00-496f-a41e-1d7f2a294a5b-plugins-config-volume\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:27:30.535590 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:30.535544 6960 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3541c21-2190-4ba9-a2aa-286582db6f34-plugins-config-volume\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:27:30.537225 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:30.537203 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ebea224-1a00-496f-a41e-1d7f2a294a5b-kube-api-access-nmmgh" pod "6ebea224-1a00-496f-a41e-1d7f2a294a5b" (UID: "6ebea224-1a00-496f-a41e-1d7f2a294a5b"). InnerVolumeSpecName "kube-api-access-nmmgh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:27:30.537225 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:30.537218 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3541c21-2190-4ba9-a2aa-286582db6f34-kube-api-access-8fgps" pod "f3541c21-2190-4ba9-a2aa-286582db6f34" (UID: "f3541c21-2190-4ba9-a2aa-286582db6f34"). InnerVolumeSpecName "kube-api-access-8fgps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:27:30.635983 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:30.635960 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmmgh\" (UniqueName: \"kubernetes.io/projected/6ebea224-1a00-496f-a41e-1d7f2a294a5b-kube-api-access-nmmgh\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:27:30.635983 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:30.635983 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fgps\" (UniqueName: \"kubernetes.io/projected/f3541c21-2190-4ba9-a2aa-286582db6f34-kube-api-access-8fgps\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:27:30.685694 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:30.685663 6960 generic.go:356] "Generic (PLEG): container finished" podID="f3541c21-2190-4ba9-a2aa-286582db6f34" containerID="18b412ec06a44da429e002741338dc60551bb66300e93211144a06ae621d4680" exitCode=0 Oct 30 19:27:30.685829 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:30.685709 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b0e009-bfd774574-dl2n4" Oct 30 19:27:30.685829 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:30.685746 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b0e009-bfd774574-dl2n4" event={"ID":"f3541c21-2190-4ba9-a2aa-286582db6f34","Type":"ContainerDied","Data":"18b412ec06a44da429e002741338dc60551bb66300e93211144a06ae621d4680"} Oct 30 19:27:30.685829 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:30.685796 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b0e009-bfd774574-dl2n4" event={"ID":"f3541c21-2190-4ba9-a2aa-286582db6f34","Type":"ContainerDied","Data":"1215f5cbcf060ffb529686f3bab7c44d3579f5c34bd6c102e4c4cfc15016e1a4"} Oct 30 19:27:30.685829 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:30.685824 6960 scope.go:122] "RemoveContainer" containerID="18b412ec06a44da429e002741338dc60551bb66300e93211144a06ae621d4680" Oct 30 19:27:30.687400 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:30.687373 6960 generic.go:356] "Generic (PLEG): container finished" podID="6ebea224-1a00-496f-a41e-1d7f2a294a5b" containerID="797f04cb0740af078454d1f763dbfcf7ebf5c88bd63029ccdd5bf0e4fbe1dba7" exitCode=0 Oct 30 19:27:30.687514 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:30.687419 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b0e009-bb65989f9-m7q2f" Oct 30 19:27:30.687514 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:30.687419 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b0e009-bb65989f9-m7q2f" event={"ID":"6ebea224-1a00-496f-a41e-1d7f2a294a5b","Type":"ContainerDied","Data":"797f04cb0740af078454d1f763dbfcf7ebf5c88bd63029ccdd5bf0e4fbe1dba7"} Oct 30 19:27:30.687514 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:30.687469 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b0e009-bb65989f9-m7q2f" event={"ID":"6ebea224-1a00-496f-a41e-1d7f2a294a5b","Type":"ContainerDied","Data":"019746db4dc6fb1ffb02be63955ffaafbe83fd18e29b1552911aedef6e15024c"} Oct 30 19:27:30.695222 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:30.695208 6960 scope.go:122] "RemoveContainer" containerID="18b412ec06a44da429e002741338dc60551bb66300e93211144a06ae621d4680" Oct 30 19:27:30.695456 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:27:30.695436 6960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18b412ec06a44da429e002741338dc60551bb66300e93211144a06ae621d4680\": container with ID starting with 18b412ec06a44da429e002741338dc60551bb66300e93211144a06ae621d4680 not found: ID does not exist" containerID="18b412ec06a44da429e002741338dc60551bb66300e93211144a06ae621d4680" Oct 30 19:27:30.695510 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:30.695462 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18b412ec06a44da429e002741338dc60551bb66300e93211144a06ae621d4680"} err="failed to get container status \"18b412ec06a44da429e002741338dc60551bb66300e93211144a06ae621d4680\": rpc error: code = NotFound desc = could not find container \"18b412ec06a44da429e002741338dc60551bb66300e93211144a06ae621d4680\": container with ID starting with 18b412ec06a44da429e002741338dc60551bb66300e93211144a06ae621d4680 not found: ID does not exist" Oct 30 19:27:30.695510 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:30.695484 6960 scope.go:122] "RemoveContainer" containerID="797f04cb0740af078454d1f763dbfcf7ebf5c88bd63029ccdd5bf0e4fbe1dba7" Oct 30 19:27:30.703618 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:30.703601 6960 scope.go:122] "RemoveContainer" containerID="797f04cb0740af078454d1f763dbfcf7ebf5c88bd63029ccdd5bf0e4fbe1dba7" Oct 30 19:27:30.703899 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:27:30.703879 6960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"797f04cb0740af078454d1f763dbfcf7ebf5c88bd63029ccdd5bf0e4fbe1dba7\": container with ID starting with 797f04cb0740af078454d1f763dbfcf7ebf5c88bd63029ccdd5bf0e4fbe1dba7 not found: ID does not exist" containerID="797f04cb0740af078454d1f763dbfcf7ebf5c88bd63029ccdd5bf0e4fbe1dba7" Oct 30 19:27:30.703985 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:30.703904 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"797f04cb0740af078454d1f763dbfcf7ebf5c88bd63029ccdd5bf0e4fbe1dba7"} err="failed to get container status \"797f04cb0740af078454d1f763dbfcf7ebf5c88bd63029ccdd5bf0e4fbe1dba7\": rpc error: code = NotFound desc = could not find container \"797f04cb0740af078454d1f763dbfcf7ebf5c88bd63029ccdd5bf0e4fbe1dba7\": container with ID starting with 797f04cb0740af078454d1f763dbfcf7ebf5c88bd63029ccdd5bf0e4fbe1dba7 not found: ID does not exist" Oct 30 19:27:30.705788 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:30.705768 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b0e009-bb65989f9-m7q2f"] Oct 30 19:27:30.710214 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:30.710194 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b0e009-bb65989f9-m7q2f"] Oct 30 19:27:30.724632 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:30.722217 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-b0e009-bfd774574-dl2n4"] Oct 30 19:27:30.729604 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:30.729580 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-b0e009-bfd774574-dl2n4"] Oct 30 19:27:30.911810 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:30.911747 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:31.725165 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:31.725119 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:32.544255 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:32.544199 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:32.591560 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:32.591529 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ebea224-1a00-496f-a41e-1d7f2a294a5b" path="/var/lib/kubelet/pods/6ebea224-1a00-496f-a41e-1d7f2a294a5b/volumes" Oct 30 19:27:32.591925 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:32.591913 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3541c21-2190-4ba9-a2aa-286582db6f34" path="/var/lib/kubelet/pods/f3541c21-2190-4ba9-a2aa-286582db6f34/volumes" Oct 30 19:27:33.337078 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:33.337047 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:34.126690 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:34.126654 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:34.864292 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:34.864255 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:34.986455 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:34.986418 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:35.249611 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:35.249527 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:35.705789 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:35.705748 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:36.124557 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:36.124522 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:36.916121 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:36.916081 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:37.715832 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:37.715784 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:38.528759 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:38.528721 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:39.416574 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:39.416542 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:40.224503 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:40.224466 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:40.654339 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:40.654308 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-8a92cc/distinct-2-w2-8a92cc-kserve-6cc4b9c974-n7cd2"] Oct 30 19:27:40.681427 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:40.681395 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-8a92cc/simulator-w2-8a92cc-kserve-fb77d84bc-nqf9q"] Oct 30 19:27:40.778115 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:40.778079 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:41.111030 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:41.110985 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:41.912862 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:41.912829 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:42.716452 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:42.716406 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:43.509155 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:43.509084 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:44.317652 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:44.317620 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:45.330126 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:45.330088 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:46.209294 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:46.209256 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:46.327203 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:46.327164 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:53.933818 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:27:53.933780 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 19:27:55.003310 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:55.003277 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:27:55.675526 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:55.675486 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-2-w2-8a92cc-kserve-6cc4b9c974-n7cd2" podUID="e5fd8f33-8662-4f90-9817-e601f694bf1d" containerName="main" containerID="cri-o://4fc2469d4f9c0f84d2cfad935cfca3034ebf064a33e3a578ffa150f56a040cf9" gracePeriod=45 Oct 30 19:27:55.702087 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:55.702059 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-8a92cc/simulator-w2-8a92cc-kserve-fb77d84bc-nqf9q" podUID="ce70492e-c727-4dc1-b2ee-bf278d29a45b" containerName="main" containerID="cri-o://d0e5564d705a626eda818f4877ab018aa9b256cd9a24f804b3ffad218b6c744a" gracePeriod=45 Oct 30 19:27:55.800905 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:55.800872 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8a92cc/simulator-w2-8a92cc-kserve-fb77d84bc-nqf9q" podUID="ce70492e-c727-4dc1-b2ee-bf278d29a45b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": dial tcp 10.134.0.48:8000: connect: connection refused" Oct 30 19:27:55.829473 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:55.829445 6960 generic.go:356] "Generic (PLEG): container finished" podID="e5fd8f33-8662-4f90-9817-e601f694bf1d" containerID="4fc2469d4f9c0f84d2cfad935cfca3034ebf064a33e3a578ffa150f56a040cf9" exitCode=0 Oct 30 19:27:55.829634 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:55.829518 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-2-w2-8a92cc-kserve-6cc4b9c974-n7cd2" event={"ID":"e5fd8f33-8662-4f90-9817-e601f694bf1d","Type":"ContainerDied","Data":"4fc2469d4f9c0f84d2cfad935cfca3034ebf064a33e3a578ffa150f56a040cf9"} Oct 30 19:27:55.831497 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:55.831467 6960 generic.go:356] "Generic (PLEG): container finished" podID="ce70492e-c727-4dc1-b2ee-bf278d29a45b" containerID="d0e5564d705a626eda818f4877ab018aa9b256cd9a24f804b3ffad218b6c744a" exitCode=0 Oct 30 19:27:55.831662 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:55.831540 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-8a92cc/simulator-w2-8a92cc-kserve-fb77d84bc-nqf9q" event={"ID":"ce70492e-c727-4dc1-b2ee-bf278d29a45b","Type":"ContainerDied","Data":"d0e5564d705a626eda818f4877ab018aa9b256cd9a24f804b3ffad218b6c744a"} Oct 30 19:27:55.956825 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:55.956800 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-2-w2-8a92cc-kserve-6cc4b9c974-n7cd2" Oct 30 19:27:55.990668 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:55.990640 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-8a92cc/simulator-w2-8a92cc-kserve-fb77d84bc-nqf9q" Oct 30 19:27:56.090136 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:56.090104 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce70492e-c727-4dc1-b2ee-bf278d29a45b-kube-api-access-4knng\" (UniqueName: \"kubernetes.io/projected/ce70492e-c727-4dc1-b2ee-bf278d29a45b-kube-api-access-4knng\") pod \"ce70492e-c727-4dc1-b2ee-bf278d29a45b\" (UID: \"ce70492e-c727-4dc1-b2ee-bf278d29a45b\") " Oct 30 19:27:56.090591 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:56.090180 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5fd8f33-8662-4f90-9817-e601f694bf1d-tls-certs\" (UniqueName: \"kubernetes.io/secret/e5fd8f33-8662-4f90-9817-e601f694bf1d-tls-certs\") pod \"e5fd8f33-8662-4f90-9817-e601f694bf1d\" (UID: \"e5fd8f33-8662-4f90-9817-e601f694bf1d\") " Oct 30 19:27:56.090591 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:56.090241 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce70492e-c727-4dc1-b2ee-bf278d29a45b-home\" (UniqueName: \"kubernetes.io/empty-dir/ce70492e-c727-4dc1-b2ee-bf278d29a45b-home\") pod \"ce70492e-c727-4dc1-b2ee-bf278d29a45b\" (UID: \"ce70492e-c727-4dc1-b2ee-bf278d29a45b\") " Oct 30 19:27:56.090591 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:56.090303 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5fd8f33-8662-4f90-9817-e601f694bf1d-home\" (UniqueName: \"kubernetes.io/empty-dir/e5fd8f33-8662-4f90-9817-e601f694bf1d-home\") pod \"e5fd8f33-8662-4f90-9817-e601f694bf1d\" (UID: \"e5fd8f33-8662-4f90-9817-e601f694bf1d\") " Oct 30 19:27:56.090591 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:56.090347 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5fd8f33-8662-4f90-9817-e601f694bf1d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5fd8f33-8662-4f90-9817-e601f694bf1d-model-cache\") pod \"e5fd8f33-8662-4f90-9817-e601f694bf1d\" (UID: \"e5fd8f33-8662-4f90-9817-e601f694bf1d\") " Oct 30 19:27:56.090591 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:56.090381 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce70492e-c727-4dc1-b2ee-bf278d29a45b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce70492e-c727-4dc1-b2ee-bf278d29a45b-tmp-dir\") pod \"ce70492e-c727-4dc1-b2ee-bf278d29a45b\" (UID: \"ce70492e-c727-4dc1-b2ee-bf278d29a45b\") " Oct 30 19:27:56.090591 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:56.090412 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce70492e-c727-4dc1-b2ee-bf278d29a45b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce70492e-c727-4dc1-b2ee-bf278d29a45b-model-cache\") pod \"ce70492e-c727-4dc1-b2ee-bf278d29a45b\" (UID: \"ce70492e-c727-4dc1-b2ee-bf278d29a45b\") " Oct 30 19:27:56.090591 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:56.090452 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce70492e-c727-4dc1-b2ee-bf278d29a45b-tls-certs\" (UniqueName: \"kubernetes.io/secret/ce70492e-c727-4dc1-b2ee-bf278d29a45b-tls-certs\") pod \"ce70492e-c727-4dc1-b2ee-bf278d29a45b\" (UID: \"ce70492e-c727-4dc1-b2ee-bf278d29a45b\") " Oct 30 19:27:56.090591 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:56.090499 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5fd8f33-8662-4f90-9817-e601f694bf1d-kube-api-access-pbsd8\" (UniqueName: \"kubernetes.io/projected/e5fd8f33-8662-4f90-9817-e601f694bf1d-kube-api-access-pbsd8\") pod \"e5fd8f33-8662-4f90-9817-e601f694bf1d\" (UID: \"e5fd8f33-8662-4f90-9817-e601f694bf1d\") " Oct 30 19:27:56.090591 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:56.090537 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5fd8f33-8662-4f90-9817-e601f694bf1d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5fd8f33-8662-4f90-9817-e601f694bf1d-dshm\") pod \"e5fd8f33-8662-4f90-9817-e601f694bf1d\" (UID: \"e5fd8f33-8662-4f90-9817-e601f694bf1d\") " Oct 30 19:27:56.090591 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:56.090557 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce70492e-c727-4dc1-b2ee-bf278d29a45b-home" pod "ce70492e-c727-4dc1-b2ee-bf278d29a45b" (UID: "ce70492e-c727-4dc1-b2ee-bf278d29a45b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:27:56.090591 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:56.090582 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5fd8f33-8662-4f90-9817-e601f694bf1d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5fd8f33-8662-4f90-9817-e601f694bf1d-tmp-dir\") pod \"e5fd8f33-8662-4f90-9817-e601f694bf1d\" (UID: \"e5fd8f33-8662-4f90-9817-e601f694bf1d\") " Oct 30 19:27:56.091236 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:56.090590 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5fd8f33-8662-4f90-9817-e601f694bf1d-home" pod "e5fd8f33-8662-4f90-9817-e601f694bf1d" (UID: "e5fd8f33-8662-4f90-9817-e601f694bf1d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:27:56.091236 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:56.090626 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce70492e-c727-4dc1-b2ee-bf278d29a45b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce70492e-c727-4dc1-b2ee-bf278d29a45b-dshm\") pod \"ce70492e-c727-4dc1-b2ee-bf278d29a45b\" (UID: \"ce70492e-c727-4dc1-b2ee-bf278d29a45b\") " Oct 30 19:27:56.091236 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:56.090737 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce70492e-c727-4dc1-b2ee-bf278d29a45b-model-cache" pod "ce70492e-c727-4dc1-b2ee-bf278d29a45b" (UID: "ce70492e-c727-4dc1-b2ee-bf278d29a45b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:27:56.091236 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:56.090832 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce70492e-c727-4dc1-b2ee-bf278d29a45b-tmp-dir" pod "ce70492e-c727-4dc1-b2ee-bf278d29a45b" (UID: "ce70492e-c727-4dc1-b2ee-bf278d29a45b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:27:56.091236 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:56.090973 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5fd8f33-8662-4f90-9817-e601f694bf1d-model-cache" pod "e5fd8f33-8662-4f90-9817-e601f694bf1d" (UID: "e5fd8f33-8662-4f90-9817-e601f694bf1d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:27:56.091236 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:56.091186 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5fd8f33-8662-4f90-9817-e601f694bf1d-tmp-dir" pod "e5fd8f33-8662-4f90-9817-e601f694bf1d" (UID: "e5fd8f33-8662-4f90-9817-e601f694bf1d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:27:56.091236 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:56.091205 6960 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce70492e-c727-4dc1-b2ee-bf278d29a45b-home\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:27:56.091236 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:56.091224 6960 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5fd8f33-8662-4f90-9817-e601f694bf1d-home\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:27:56.091236 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:56.091238 6960 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5fd8f33-8662-4f90-9817-e601f694bf1d-model-cache\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:27:56.091909 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:56.091252 6960 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce70492e-c727-4dc1-b2ee-bf278d29a45b-tmp-dir\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:27:56.091909 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:56.091269 6960 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce70492e-c727-4dc1-b2ee-bf278d29a45b-model-cache\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:27:56.092989 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:56.092964 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5fd8f33-8662-4f90-9817-e601f694bf1d-tls-certs" pod "e5fd8f33-8662-4f90-9817-e601f694bf1d" (UID: "e5fd8f33-8662-4f90-9817-e601f694bf1d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:27:56.093285 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:56.093265 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce70492e-c727-4dc1-b2ee-bf278d29a45b-kube-api-access-4knng" pod "ce70492e-c727-4dc1-b2ee-bf278d29a45b" (UID: "ce70492e-c727-4dc1-b2ee-bf278d29a45b"). InnerVolumeSpecName "kube-api-access-4knng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:27:56.093361 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:56.093347 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce70492e-c727-4dc1-b2ee-bf278d29a45b-tls-certs" pod "ce70492e-c727-4dc1-b2ee-bf278d29a45b" (UID: "ce70492e-c727-4dc1-b2ee-bf278d29a45b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:27:56.093456 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:56.093436 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5fd8f33-8662-4f90-9817-e601f694bf1d-dshm" pod "e5fd8f33-8662-4f90-9817-e601f694bf1d" (UID: "e5fd8f33-8662-4f90-9817-e601f694bf1d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:27:56.093551 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:56.093537 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce70492e-c727-4dc1-b2ee-bf278d29a45b-dshm" pod "ce70492e-c727-4dc1-b2ee-bf278d29a45b" (UID: "ce70492e-c727-4dc1-b2ee-bf278d29a45b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:27:56.093719 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:56.093698 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5fd8f33-8662-4f90-9817-e601f694bf1d-kube-api-access-pbsd8" pod "e5fd8f33-8662-4f90-9817-e601f694bf1d" (UID: "e5fd8f33-8662-4f90-9817-e601f694bf1d"). InnerVolumeSpecName "kube-api-access-pbsd8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:27:56.191853 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:56.191769 6960 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce70492e-c727-4dc1-b2ee-bf278d29a45b-tls-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:27:56.191853 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:56.191794 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbsd8\" (UniqueName: \"kubernetes.io/projected/e5fd8f33-8662-4f90-9817-e601f694bf1d-kube-api-access-pbsd8\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:27:56.191853 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:56.191803 6960 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5fd8f33-8662-4f90-9817-e601f694bf1d-dshm\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:27:56.191853 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:56.191812 6960 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5fd8f33-8662-4f90-9817-e601f694bf1d-tmp-dir\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:27:56.191853 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:56.191819 6960 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce70492e-c727-4dc1-b2ee-bf278d29a45b-dshm\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:27:56.191853 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:56.191828 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4knng\" (UniqueName: \"kubernetes.io/projected/ce70492e-c727-4dc1-b2ee-bf278d29a45b-kube-api-access-4knng\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:27:56.191853 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:56.191838 6960 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5fd8f33-8662-4f90-9817-e601f694bf1d-tls-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:27:56.838381 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:56.838354 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-2-w2-8a92cc-kserve-6cc4b9c974-n7cd2" Oct 30 19:27:56.838559 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:56.838353 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-8a92cc/distinct-2-w2-8a92cc-kserve-6cc4b9c974-n7cd2" event={"ID":"e5fd8f33-8662-4f90-9817-e601f694bf1d","Type":"ContainerDied","Data":"a2d176f3b8cc98642931f3eabb2f683a1e7b80f369c28aa1e9f3a02b3bb5fe82"} Oct 30 19:27:56.838559 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:56.838485 6960 scope.go:122] "RemoveContainer" containerID="4fc2469d4f9c0f84d2cfad935cfca3034ebf064a33e3a578ffa150f56a040cf9" Oct 30 19:27:56.839870 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:56.839845 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-8a92cc/simulator-w2-8a92cc-kserve-fb77d84bc-nqf9q" Oct 30 19:27:56.839870 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:56.839845 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-8a92cc/simulator-w2-8a92cc-kserve-fb77d84bc-nqf9q" event={"ID":"ce70492e-c727-4dc1-b2ee-bf278d29a45b","Type":"ContainerDied","Data":"081f4df771a95056f467678a17917fedb56dc38bdb907922da70305c1f5d968e"} Oct 30 19:27:56.847249 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:56.847228 6960 scope.go:122] "RemoveContainer" containerID="d0e5564d705a626eda818f4877ab018aa9b256cd9a24f804b3ffad218b6c744a" Oct 30 19:27:56.853507 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:56.853455 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zldp5"] Oct 30 19:27:56.853648 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:56.853629 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zldp5" podUID="d12464a9-d9c7-46e6-b3cc-ab0c16d75070" containerName="istio-proxy" containerID="cri-o://a1a996452ec360585f099b43945dfb0ba14d0909c89217ebd479c58f07816315" gracePeriod=30 Oct 30 19:27:56.858266 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:56.858248 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-8a92cc/simulator-w2-8a92cc-kserve-fb77d84bc-nqf9q"] Oct 30 19:27:56.870281 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:56.870260 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-8a92cc/simulator-w2-8a92cc-kserve-fb77d84bc-nqf9q"] Oct 30 19:27:56.881377 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:56.881355 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-8a92cc/distinct-2-w2-8a92cc-kserve-6cc4b9c974-n7cd2"] Oct 30 19:27:56.889611 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:56.889586 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-8a92cc/distinct-2-w2-8a92cc-kserve-6cc4b9c974-n7cd2"] Oct 30 19:27:58.591632 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:58.591596 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce70492e-c727-4dc1-b2ee-bf278d29a45b" path="/var/lib/kubelet/pods/ce70492e-c727-4dc1-b2ee-bf278d29a45b/volumes" Oct 30 19:27:58.592098 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:27:58.592007 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5fd8f33-8662-4f90-9817-e601f694bf1d" path="/var/lib/kubelet/pods/e5fd8f33-8662-4f90-9817-e601f694bf1d/volumes" Oct 30 19:28:01.964070 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:01.964037 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-8a92cc-788c8c9c4b-6gtz2"] Oct 30 19:28:01.964671 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:01.964302 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8a92cc-788c8c9c4b-6gtz2" podUID="7998c864-d4ca-4b59-9ef6-0838726a0c8c" containerName="maas-api" containerID="cri-o://90dbdc8f6ec52b6ee54d815672417459eb3ecc43e61b3c33fb9dd0135428c6a8" gracePeriod=30 Oct 30 19:28:02.033793 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:02.033761 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:02.138810 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:02.137641 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zldp5" Oct 30 19:28:02.238878 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:02.238842 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8a92cc-788c8c9c4b-6gtz2" Oct 30 19:28:02.251415 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:02.251386 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d12464a9-d9c7-46e6-b3cc-ab0c16d75070-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d12464a9-d9c7-46e6-b3cc-ab0c16d75070-istiod-ca-cert\") pod \"d12464a9-d9c7-46e6-b3cc-ab0c16d75070\" (UID: \"d12464a9-d9c7-46e6-b3cc-ab0c16d75070\") " Oct 30 19:28:02.251589 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:02.251409 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:02.251589 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:02.251428 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d12464a9-d9c7-46e6-b3cc-ab0c16d75070-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d12464a9-d9c7-46e6-b3cc-ab0c16d75070-istio-data\") pod \"d12464a9-d9c7-46e6-b3cc-ab0c16d75070\" (UID: \"d12464a9-d9c7-46e6-b3cc-ab0c16d75070\") " Oct 30 19:28:02.251589 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:02.251456 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d12464a9-d9c7-46e6-b3cc-ab0c16d75070-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d12464a9-d9c7-46e6-b3cc-ab0c16d75070-workload-socket\") pod \"d12464a9-d9c7-46e6-b3cc-ab0c16d75070\" (UID: \"d12464a9-d9c7-46e6-b3cc-ab0c16d75070\") " Oct 30 19:28:02.251589 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:02.251544 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d12464a9-d9c7-46e6-b3cc-ab0c16d75070-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d12464a9-d9c7-46e6-b3cc-ab0c16d75070-workload-certs\") pod \"d12464a9-d9c7-46e6-b3cc-ab0c16d75070\" (UID: \"d12464a9-d9c7-46e6-b3cc-ab0c16d75070\") " Oct 30 19:28:02.251589 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:02.251571 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d12464a9-d9c7-46e6-b3cc-ab0c16d75070-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d12464a9-d9c7-46e6-b3cc-ab0c16d75070-istio-envoy\") pod \"d12464a9-d9c7-46e6-b3cc-ab0c16d75070\" (UID: \"d12464a9-d9c7-46e6-b3cc-ab0c16d75070\") " Oct 30 19:28:02.251929 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:02.251603 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d12464a9-d9c7-46e6-b3cc-ab0c16d75070-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d12464a9-d9c7-46e6-b3cc-ab0c16d75070-istio-podinfo\") pod \"d12464a9-d9c7-46e6-b3cc-ab0c16d75070\" (UID: \"d12464a9-d9c7-46e6-b3cc-ab0c16d75070\") " Oct 30 19:28:02.251929 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:02.251643 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d12464a9-d9c7-46e6-b3cc-ab0c16d75070-kube-api-access-95pld\" (UniqueName: \"kubernetes.io/projected/d12464a9-d9c7-46e6-b3cc-ab0c16d75070-kube-api-access-95pld\") pod \"d12464a9-d9c7-46e6-b3cc-ab0c16d75070\" (UID: \"d12464a9-d9c7-46e6-b3cc-ab0c16d75070\") " Oct 30 19:28:02.251929 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:02.251672 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d12464a9-d9c7-46e6-b3cc-ab0c16d75070-istio-token\" (UniqueName: \"kubernetes.io/projected/d12464a9-d9c7-46e6-b3cc-ab0c16d75070-istio-token\") pod \"d12464a9-d9c7-46e6-b3cc-ab0c16d75070\" (UID: \"d12464a9-d9c7-46e6-b3cc-ab0c16d75070\") " Oct 30 19:28:02.251929 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:02.251710 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d12464a9-d9c7-46e6-b3cc-ab0c16d75070-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d12464a9-d9c7-46e6-b3cc-ab0c16d75070-credential-socket\") pod \"d12464a9-d9c7-46e6-b3cc-ab0c16d75070\" (UID: \"d12464a9-d9c7-46e6-b3cc-ab0c16d75070\") " Oct 30 19:28:02.251929 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:02.251791 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d12464a9-d9c7-46e6-b3cc-ab0c16d75070-istiod-ca-cert" pod "d12464a9-d9c7-46e6-b3cc-ab0c16d75070" (UID: "d12464a9-d9c7-46e6-b3cc-ab0c16d75070"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:28:02.252268 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:02.252000 6960 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d12464a9-d9c7-46e6-b3cc-ab0c16d75070-istiod-ca-cert\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:28:02.252335 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:02.252284 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d12464a9-d9c7-46e6-b3cc-ab0c16d75070-workload-socket" pod "d12464a9-d9c7-46e6-b3cc-ab0c16d75070" (UID: "d12464a9-d9c7-46e6-b3cc-ab0c16d75070"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:28:02.252444 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:02.252422 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d12464a9-d9c7-46e6-b3cc-ab0c16d75070-workload-certs" pod "d12464a9-d9c7-46e6-b3cc-ab0c16d75070" (UID: "d12464a9-d9c7-46e6-b3cc-ab0c16d75070"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:28:02.252641 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:02.252618 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d12464a9-d9c7-46e6-b3cc-ab0c16d75070-credential-socket" pod "d12464a9-d9c7-46e6-b3cc-ab0c16d75070" (UID: "d12464a9-d9c7-46e6-b3cc-ab0c16d75070"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:28:02.253015 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:02.252986 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d12464a9-d9c7-46e6-b3cc-ab0c16d75070-istio-data" pod "d12464a9-d9c7-46e6-b3cc-ab0c16d75070" (UID: "d12464a9-d9c7-46e6-b3cc-ab0c16d75070"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:28:02.254287 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:02.254259 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d12464a9-d9c7-46e6-b3cc-ab0c16d75070-istio-envoy" pod "d12464a9-d9c7-46e6-b3cc-ab0c16d75070" (UID: "d12464a9-d9c7-46e6-b3cc-ab0c16d75070"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:28:02.254554 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:02.254534 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d12464a9-d9c7-46e6-b3cc-ab0c16d75070-istio-token" pod "d12464a9-d9c7-46e6-b3cc-ab0c16d75070" (UID: "d12464a9-d9c7-46e6-b3cc-ab0c16d75070"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:28:02.255314 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:02.255287 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d12464a9-d9c7-46e6-b3cc-ab0c16d75070-istio-podinfo" pod "d12464a9-d9c7-46e6-b3cc-ab0c16d75070" (UID: "d12464a9-d9c7-46e6-b3cc-ab0c16d75070"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:28:02.257787 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:02.257764 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d12464a9-d9c7-46e6-b3cc-ab0c16d75070-kube-api-access-95pld" pod "d12464a9-d9c7-46e6-b3cc-ab0c16d75070" (UID: "d12464a9-d9c7-46e6-b3cc-ab0c16d75070"). InnerVolumeSpecName "kube-api-access-95pld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:28:02.353010 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:02.352982 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7998c864-d4ca-4b59-9ef6-0838726a0c8c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7998c864-d4ca-4b59-9ef6-0838726a0c8c-openshift-service-ca\") pod \"7998c864-d4ca-4b59-9ef6-0838726a0c8c\" (UID: \"7998c864-d4ca-4b59-9ef6-0838726a0c8c\") " Oct 30 19:28:02.353181 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:02.353033 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7998c864-d4ca-4b59-9ef6-0838726a0c8c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7998c864-d4ca-4b59-9ef6-0838726a0c8c-metrics-certs\") pod \"7998c864-d4ca-4b59-9ef6-0838726a0c8c\" (UID: \"7998c864-d4ca-4b59-9ef6-0838726a0c8c\") " Oct 30 19:28:02.353181 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:02.353053 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7998c864-d4ca-4b59-9ef6-0838726a0c8c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7998c864-d4ca-4b59-9ef6-0838726a0c8c-maas-api-tls\") pod \"7998c864-d4ca-4b59-9ef6-0838726a0c8c\" (UID: \"7998c864-d4ca-4b59-9ef6-0838726a0c8c\") " Oct 30 19:28:02.353181 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:02.353083 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7998c864-d4ca-4b59-9ef6-0838726a0c8c-kube-api-access-ggls4\" (UniqueName: \"kubernetes.io/projected/7998c864-d4ca-4b59-9ef6-0838726a0c8c-kube-api-access-ggls4\") pod \"7998c864-d4ca-4b59-9ef6-0838726a0c8c\" (UID: \"7998c864-d4ca-4b59-9ef6-0838726a0c8c\") " Oct 30 19:28:02.353404 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:02.353379 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7998c864-d4ca-4b59-9ef6-0838726a0c8c-openshift-service-ca" pod "7998c864-d4ca-4b59-9ef6-0838726a0c8c" (UID: "7998c864-d4ca-4b59-9ef6-0838726a0c8c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:28:02.353502 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:02.353485 6960 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d12464a9-d9c7-46e6-b3cc-ab0c16d75070-credential-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:28:02.353547 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:02.353508 6960 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7998c864-d4ca-4b59-9ef6-0838726a0c8c-openshift-service-ca\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:28:02.353547 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:02.353523 6960 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d12464a9-d9c7-46e6-b3cc-ab0c16d75070-istio-data\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:28:02.353547 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:02.353533 6960 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d12464a9-d9c7-46e6-b3cc-ab0c16d75070-workload-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:28:02.353547 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:02.353542 6960 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d12464a9-d9c7-46e6-b3cc-ab0c16d75070-workload-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:28:02.353717 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:02.353552 6960 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d12464a9-d9c7-46e6-b3cc-ab0c16d75070-istio-envoy\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:28:02.353717 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:02.353561 6960 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d12464a9-d9c7-46e6-b3cc-ab0c16d75070-istio-podinfo\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:28:02.353717 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:02.353571 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95pld\" (UniqueName: \"kubernetes.io/projected/d12464a9-d9c7-46e6-b3cc-ab0c16d75070-kube-api-access-95pld\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:28:02.353717 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:02.353580 6960 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d12464a9-d9c7-46e6-b3cc-ab0c16d75070-istio-token\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:28:02.355253 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:02.355229 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7998c864-d4ca-4b59-9ef6-0838726a0c8c-maas-api-tls" pod "7998c864-d4ca-4b59-9ef6-0838726a0c8c" (UID: "7998c864-d4ca-4b59-9ef6-0838726a0c8c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:28:02.355843 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:02.355822 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7998c864-d4ca-4b59-9ef6-0838726a0c8c-metrics-certs" pod "7998c864-d4ca-4b59-9ef6-0838726a0c8c" (UID: "7998c864-d4ca-4b59-9ef6-0838726a0c8c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:28:02.355914 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:02.355842 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7998c864-d4ca-4b59-9ef6-0838726a0c8c-kube-api-access-ggls4" pod "7998c864-d4ca-4b59-9ef6-0838726a0c8c" (UID: "7998c864-d4ca-4b59-9ef6-0838726a0c8c"). InnerVolumeSpecName "kube-api-access-ggls4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:28:02.360425 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:02.360397 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:02.454360 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:02.454287 6960 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7998c864-d4ca-4b59-9ef6-0838726a0c8c-metrics-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:28:02.454360 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:02.454312 6960 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7998c864-d4ca-4b59-9ef6-0838726a0c8c-maas-api-tls\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:28:02.454360 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:02.454323 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggls4\" (UniqueName: \"kubernetes.io/projected/7998c864-d4ca-4b59-9ef6-0838726a0c8c-kube-api-access-ggls4\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:28:02.789024 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:02.788933 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:02.872597 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:02.872567 6960 generic.go:356] "Generic (PLEG): container finished" podID="7998c864-d4ca-4b59-9ef6-0838726a0c8c" containerID="90dbdc8f6ec52b6ee54d815672417459eb3ecc43e61b3c33fb9dd0135428c6a8" exitCode=0 Oct 30 19:28:02.872789 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:02.872622 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8a92cc-788c8c9c4b-6gtz2" Oct 30 19:28:02.872789 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:02.872642 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8a92cc-788c8c9c4b-6gtz2" event={"ID":"7998c864-d4ca-4b59-9ef6-0838726a0c8c","Type":"ContainerDied","Data":"90dbdc8f6ec52b6ee54d815672417459eb3ecc43e61b3c33fb9dd0135428c6a8"} Oct 30 19:28:02.872789 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:02.872690 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8a92cc-788c8c9c4b-6gtz2" event={"ID":"7998c864-d4ca-4b59-9ef6-0838726a0c8c","Type":"ContainerDied","Data":"c33340675be6901d90c104660b69fa5c59e5a32f230341166c920303814cc9d9"} Oct 30 19:28:02.872789 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:02.872712 6960 scope.go:122] "RemoveContainer" containerID="90dbdc8f6ec52b6ee54d815672417459eb3ecc43e61b3c33fb9dd0135428c6a8" Oct 30 19:28:02.874171 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:02.874110 6960 generic.go:356] "Generic (PLEG): container finished" podID="d12464a9-d9c7-46e6-b3cc-ab0c16d75070" containerID="a1a996452ec360585f099b43945dfb0ba14d0909c89217ebd479c58f07816315" exitCode=0 Oct 30 19:28:02.874289 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:02.874176 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zldp5" event={"ID":"d12464a9-d9c7-46e6-b3cc-ab0c16d75070","Type":"ContainerDied","Data":"a1a996452ec360585f099b43945dfb0ba14d0909c89217ebd479c58f07816315"} Oct 30 19:28:02.874289 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:02.874242 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zldp5" event={"ID":"d12464a9-d9c7-46e6-b3cc-ab0c16d75070","Type":"ContainerDied","Data":"db5b1e2913586a856be066abfd6ffaf8b58942454fc7b23b9de4fd80b5439928"} Oct 30 19:28:02.874289 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:02.874184 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zldp5" Oct 30 19:28:02.892526 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:02.892506 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-8a92cc-788c8c9c4b-6gtz2"] Oct 30 19:28:02.893827 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:02.893811 6960 scope.go:122] "RemoveContainer" containerID="90dbdc8f6ec52b6ee54d815672417459eb3ecc43e61b3c33fb9dd0135428c6a8" Oct 30 19:28:02.894133 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:28:02.894105 6960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90dbdc8f6ec52b6ee54d815672417459eb3ecc43e61b3c33fb9dd0135428c6a8\": container with ID starting with 90dbdc8f6ec52b6ee54d815672417459eb3ecc43e61b3c33fb9dd0135428c6a8 not found: ID does not exist" containerID="90dbdc8f6ec52b6ee54d815672417459eb3ecc43e61b3c33fb9dd0135428c6a8" Oct 30 19:28:02.894281 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:02.894136 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90dbdc8f6ec52b6ee54d815672417459eb3ecc43e61b3c33fb9dd0135428c6a8"} err="failed to get container status \"90dbdc8f6ec52b6ee54d815672417459eb3ecc43e61b3c33fb9dd0135428c6a8\": rpc error: code = NotFound desc = could not find container \"90dbdc8f6ec52b6ee54d815672417459eb3ecc43e61b3c33fb9dd0135428c6a8\": container with ID starting with 90dbdc8f6ec52b6ee54d815672417459eb3ecc43e61b3c33fb9dd0135428c6a8 not found: ID does not exist" Oct 30 19:28:02.894281 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:02.894169 6960 scope.go:122] "RemoveContainer" containerID="a1a996452ec360585f099b43945dfb0ba14d0909c89217ebd479c58f07816315" Oct 30 19:28:02.896945 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:02.896924 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-8a92cc-788c8c9c4b-6gtz2"] Oct 30 19:28:02.903101 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:02.903071 6960 scope.go:122] "RemoveContainer" containerID="a1a996452ec360585f099b43945dfb0ba14d0909c89217ebd479c58f07816315" Oct 30 19:28:02.903409 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:28:02.903387 6960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1a996452ec360585f099b43945dfb0ba14d0909c89217ebd479c58f07816315\": container with ID starting with a1a996452ec360585f099b43945dfb0ba14d0909c89217ebd479c58f07816315 not found: ID does not exist" containerID="a1a996452ec360585f099b43945dfb0ba14d0909c89217ebd479c58f07816315" Oct 30 19:28:02.903489 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:02.903416 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1a996452ec360585f099b43945dfb0ba14d0909c89217ebd479c58f07816315"} err="failed to get container status \"a1a996452ec360585f099b43945dfb0ba14d0909c89217ebd479c58f07816315\": rpc error: code = NotFound desc = could not find container \"a1a996452ec360585f099b43945dfb0ba14d0909c89217ebd479c58f07816315\": container with ID starting with a1a996452ec360585f099b43945dfb0ba14d0909c89217ebd479c58f07816315 not found: ID does not exist" Oct 30 19:28:02.908090 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:02.908069 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zldp5"] Oct 30 19:28:02.912965 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:02.912939 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-zldp5"] Oct 30 19:28:04.593951 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:04.593854 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7998c864-d4ca-4b59-9ef6-0838726a0c8c" path="/var/lib/kubelet/pods/7998c864-d4ca-4b59-9ef6-0838726a0c8c/volumes" Oct 30 19:28:04.594438 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:04.594394 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d12464a9-d9c7-46e6-b3cc-ab0c16d75070" path="/var/lib/kubelet/pods/d12464a9-d9c7-46e6-b3cc-ab0c16d75070/volumes" Oct 30 19:28:09.110437 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:09.110400 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:09.221163 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:09.221109 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:09.455190 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:09.455069 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-hj7n6"] Oct 30 19:28:09.455364 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:09.455325 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-hj7n6" podUID="71c3b467-e87a-40eb-ad20-7175b0db9e49" containerName="istio-proxy" containerID="cri-o://38f11f8f890154aab95d5bb526fa1386b6f2b41ddd9140624c1c28a4afc0d9c0" gracePeriod=30 Oct 30 19:28:09.497696 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:09.497661 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:10.816591 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:10.816559 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-hm48j_a624d24b-f689-4286-9602-c89d828a0e05/manager/0.log" Oct 30 19:28:11.162061 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:11.161980 6960 patch_prober.go:29] interesting pod/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-hj7n6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:28:11.162275 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:11.162053 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-hj7n6" podUID="71c3b467-e87a-40eb-ad20-7175b0db9e49" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:28:11.187827 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:11.187804 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-f78tf_30c4e293-73c6-4d1f-8e76-6ffb15e03f51/kuadrant-console-plugin/0.log" Oct 30 19:28:12.762011 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:12.761981 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:12.866305 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:12.866272 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:13.721891 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:13.721866 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-xkxkl_13296062-8ceb-4447-9418-d52de1a4b712/manager/0.log" Oct 30 19:28:14.507700 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:14.507667 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-69dbdc8f84-kt9h9_7d18009e-009c-40be-9143-e468d1bdfc41/manager/0.log" Oct 30 19:28:14.704103 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:14.704073 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-spxts_d99c9271-c3ea-4f9b-8668-c0348c766af0/server/0.log" Oct 30 19:28:14.713480 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:14.713456 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-hj7n6" Oct 30 19:28:14.773719 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:14.773636 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71c3b467-e87a-40eb-ad20-7175b0db9e49-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71c3b467-e87a-40eb-ad20-7175b0db9e49-workload-socket\") pod \"71c3b467-e87a-40eb-ad20-7175b0db9e49\" (UID: \"71c3b467-e87a-40eb-ad20-7175b0db9e49\") " Oct 30 19:28:14.773719 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:14.773688 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71c3b467-e87a-40eb-ad20-7175b0db9e49-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71c3b467-e87a-40eb-ad20-7175b0db9e49-istio-envoy\") pod \"71c3b467-e87a-40eb-ad20-7175b0db9e49\" (UID: \"71c3b467-e87a-40eb-ad20-7175b0db9e49\") " Oct 30 19:28:14.773719 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:14.773716 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71c3b467-e87a-40eb-ad20-7175b0db9e49-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71c3b467-e87a-40eb-ad20-7175b0db9e49-istiod-ca-cert\") pod \"71c3b467-e87a-40eb-ad20-7175b0db9e49\" (UID: \"71c3b467-e87a-40eb-ad20-7175b0db9e49\") " Oct 30 19:28:14.774020 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:14.773758 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71c3b467-e87a-40eb-ad20-7175b0db9e49-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71c3b467-e87a-40eb-ad20-7175b0db9e49-credential-socket\") pod \"71c3b467-e87a-40eb-ad20-7175b0db9e49\" (UID: \"71c3b467-e87a-40eb-ad20-7175b0db9e49\") " Oct 30 19:28:14.774020 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:14.773787 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/71c3b467-e87a-40eb-ad20-7175b0db9e49-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71c3b467-e87a-40eb-ad20-7175b0db9e49-istio-podinfo\") pod \"71c3b467-e87a-40eb-ad20-7175b0db9e49\" (UID: \"71c3b467-e87a-40eb-ad20-7175b0db9e49\") " Oct 30 19:28:14.774020 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:14.773815 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71c3b467-e87a-40eb-ad20-7175b0db9e49-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71c3b467-e87a-40eb-ad20-7175b0db9e49-workload-certs\") pod \"71c3b467-e87a-40eb-ad20-7175b0db9e49\" (UID: \"71c3b467-e87a-40eb-ad20-7175b0db9e49\") " Oct 30 19:28:14.774020 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:14.773863 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71c3b467-e87a-40eb-ad20-7175b0db9e49-kube-api-access-44s46\" (UniqueName: \"kubernetes.io/projected/71c3b467-e87a-40eb-ad20-7175b0db9e49-kube-api-access-44s46\") pod \"71c3b467-e87a-40eb-ad20-7175b0db9e49\" (UID: \"71c3b467-e87a-40eb-ad20-7175b0db9e49\") " Oct 30 19:28:14.774020 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:14.773914 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71c3b467-e87a-40eb-ad20-7175b0db9e49-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71c3b467-e87a-40eb-ad20-7175b0db9e49-istio-data\") pod \"71c3b467-e87a-40eb-ad20-7175b0db9e49\" (UID: \"71c3b467-e87a-40eb-ad20-7175b0db9e49\") " Oct 30 19:28:14.774020 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:14.773944 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71c3b467-e87a-40eb-ad20-7175b0db9e49-istio-token\" (UniqueName: \"kubernetes.io/projected/71c3b467-e87a-40eb-ad20-7175b0db9e49-istio-token\") pod \"71c3b467-e87a-40eb-ad20-7175b0db9e49\" (UID: \"71c3b467-e87a-40eb-ad20-7175b0db9e49\") " Oct 30 19:28:14.774020 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:14.773961 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71c3b467-e87a-40eb-ad20-7175b0db9e49-workload-socket" pod "71c3b467-e87a-40eb-ad20-7175b0db9e49" (UID: "71c3b467-e87a-40eb-ad20-7175b0db9e49"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:28:14.774436 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:14.774128 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71c3b467-e87a-40eb-ad20-7175b0db9e49-credential-socket" pod "71c3b467-e87a-40eb-ad20-7175b0db9e49" (UID: "71c3b467-e87a-40eb-ad20-7175b0db9e49"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:28:14.774436 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:14.774176 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71c3b467-e87a-40eb-ad20-7175b0db9e49-workload-certs" pod "71c3b467-e87a-40eb-ad20-7175b0db9e49" (UID: "71c3b467-e87a-40eb-ad20-7175b0db9e49"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:28:14.774436 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:14.774258 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71c3b467-e87a-40eb-ad20-7175b0db9e49-istiod-ca-cert" pod "71c3b467-e87a-40eb-ad20-7175b0db9e49" (UID: "71c3b467-e87a-40eb-ad20-7175b0db9e49"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:28:14.774436 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:14.774349 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71c3b467-e87a-40eb-ad20-7175b0db9e49-istio-data" pod "71c3b467-e87a-40eb-ad20-7175b0db9e49" (UID: "71c3b467-e87a-40eb-ad20-7175b0db9e49"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:28:14.774436 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:14.774344 6960 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71c3b467-e87a-40eb-ad20-7175b0db9e49-credential-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:28:14.774436 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:14.774395 6960 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71c3b467-e87a-40eb-ad20-7175b0db9e49-workload-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:28:14.774436 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:14.774415 6960 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71c3b467-e87a-40eb-ad20-7175b0db9e49-workload-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:28:14.776069 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:14.776048 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71c3b467-e87a-40eb-ad20-7175b0db9e49-istio-envoy" pod "71c3b467-e87a-40eb-ad20-7175b0db9e49" (UID: "71c3b467-e87a-40eb-ad20-7175b0db9e49"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:28:14.776427 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:14.776410 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71c3b467-e87a-40eb-ad20-7175b0db9e49-istio-token" pod "71c3b467-e87a-40eb-ad20-7175b0db9e49" (UID: "71c3b467-e87a-40eb-ad20-7175b0db9e49"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:28:14.776851 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:14.776832 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71c3b467-e87a-40eb-ad20-7175b0db9e49-kube-api-access-44s46" pod "71c3b467-e87a-40eb-ad20-7175b0db9e49" (UID: "71c3b467-e87a-40eb-ad20-7175b0db9e49"). InnerVolumeSpecName "kube-api-access-44s46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:28:14.776851 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:14.776841 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/71c3b467-e87a-40eb-ad20-7175b0db9e49-istio-podinfo" pod "71c3b467-e87a-40eb-ad20-7175b0db9e49" (UID: "71c3b467-e87a-40eb-ad20-7175b0db9e49"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:28:14.875612 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:14.875572 6960 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71c3b467-e87a-40eb-ad20-7175b0db9e49-istiod-ca-cert\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:28:14.875612 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:14.875605 6960 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71c3b467-e87a-40eb-ad20-7175b0db9e49-istio-podinfo\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:28:14.875612 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:14.875616 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44s46\" (UniqueName: \"kubernetes.io/projected/71c3b467-e87a-40eb-ad20-7175b0db9e49-kube-api-access-44s46\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:28:14.875871 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:14.875627 6960 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71c3b467-e87a-40eb-ad20-7175b0db9e49-istio-data\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:28:14.875871 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:14.875635 6960 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71c3b467-e87a-40eb-ad20-7175b0db9e49-istio-token\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:28:14.875871 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:14.875645 6960 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71c3b467-e87a-40eb-ad20-7175b0db9e49-istio-envoy\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:28:14.933123 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:14.933087 6960 generic.go:356] "Generic (PLEG): container finished" podID="71c3b467-e87a-40eb-ad20-7175b0db9e49" containerID="38f11f8f890154aab95d5bb526fa1386b6f2b41ddd9140624c1c28a4afc0d9c0" exitCode=0 Oct 30 19:28:14.933330 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:14.933135 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-hj7n6" Oct 30 19:28:14.933330 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:14.933170 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-hj7n6" event={"ID":"71c3b467-e87a-40eb-ad20-7175b0db9e49","Type":"ContainerDied","Data":"38f11f8f890154aab95d5bb526fa1386b6f2b41ddd9140624c1c28a4afc0d9c0"} Oct 30 19:28:14.933330 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:14.933230 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-hj7n6" event={"ID":"71c3b467-e87a-40eb-ad20-7175b0db9e49","Type":"ContainerDied","Data":"6f3625567372ed03860620b36ebee3f07d1143f5e7f077c1bc918d628bd71df6"} Oct 30 19:28:14.933330 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:14.933253 6960 scope.go:122] "RemoveContainer" containerID="38f11f8f890154aab95d5bb526fa1386b6f2b41ddd9140624c1c28a4afc0d9c0" Oct 30 19:28:14.945189 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:14.945132 6960 scope.go:122] "RemoveContainer" containerID="38f11f8f890154aab95d5bb526fa1386b6f2b41ddd9140624c1c28a4afc0d9c0" Oct 30 19:28:14.945642 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:28:14.945621 6960 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38f11f8f890154aab95d5bb526fa1386b6f2b41ddd9140624c1c28a4afc0d9c0\": container with ID starting with 38f11f8f890154aab95d5bb526fa1386b6f2b41ddd9140624c1c28a4afc0d9c0 not found: ID does not exist" containerID="38f11f8f890154aab95d5bb526fa1386b6f2b41ddd9140624c1c28a4afc0d9c0" Oct 30 19:28:14.945757 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:14.945650 6960 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38f11f8f890154aab95d5bb526fa1386b6f2b41ddd9140624c1c28a4afc0d9c0"} err="failed to get container status \"38f11f8f890154aab95d5bb526fa1386b6f2b41ddd9140624c1c28a4afc0d9c0\": rpc error: code = NotFound desc = could not find container \"38f11f8f890154aab95d5bb526fa1386b6f2b41ddd9140624c1c28a4afc0d9c0\": container with ID starting with 38f11f8f890154aab95d5bb526fa1386b6f2b41ddd9140624c1c28a4afc0d9c0 not found: ID does not exist" Oct 30 19:28:14.956571 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:14.956540 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-hj7n6"] Oct 30 19:28:14.961012 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:14.960987 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-8a92cc-openshift-default-849c89cccc-hj7n6"] Oct 30 19:28:15.294959 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:15.294929 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-k5ljv_5b43299a-0f21-4d69-8167-6f8739c40a9b/proxy/0.log" Oct 30 19:28:16.590739 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:16.590707 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71c3b467-e87a-40eb-ad20-7175b0db9e49" path="/var/lib/kubelet/pods/71c3b467-e87a-40eb-ad20-7175b0db9e49/volumes" Oct 30 19:28:18.741695 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:18.741647 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:18.942746 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:18.942714 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:19.055196 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:19.055167 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:19.802329 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:19.802296 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:23.146583 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:23.146545 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:23.931206 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:28:23.931171 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 19:28:27.711202 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:27.711167 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-xkxkl_13296062-8ceb-4447-9418-d52de1a4b712/manager/0.log" Oct 30 19:28:28.478642 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:28.478611 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-69dbdc8f84-kt9h9_7d18009e-009c-40be-9143-e468d1bdfc41/manager/0.log" Oct 30 19:28:28.669575 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:28.669545 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-spxts_d99c9271-c3ea-4f9b-8668-c0348c766af0/server/0.log" Oct 30 19:28:29.276276 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:29.276190 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-k5ljv_5b43299a-0f21-4d69-8167-6f8739c40a9b/proxy/0.log" Oct 30 19:28:33.103660 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:33.103615 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-hm48j_a624d24b-f689-4286-9602-c89d828a0e05/manager/0.log" Oct 30 19:28:33.463550 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:33.463473 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-f78tf_30c4e293-73c6-4d1f-8e76-6ffb15e03f51/kuadrant-console-plugin/0.log" Oct 30 19:28:35.311821 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:35.311792 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68d996c55722t75_9b0ef8f8-8b08-40e1-8338-fd48c63cfcf7/istio-proxy/0.log" Oct 30 19:28:36.066595 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:36.066572 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-65c8c79996-jjcwx_fa35d05a-b5c7-40b2-ac7a-6d72ebf85a69/kube-auth-proxy/0.log" Oct 30 19:28:36.440892 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:36.440805 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-dslrt_93b3f3a7-cd85-4996-88fe-7762155b3241/istio-proxy/0.log" Oct 30 19:28:36.802344 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:36.802267 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-pb8bs_830b7d8c-c70c-4242-b81a-8be190df4aa1/istio-proxy/0.log" Oct 30 19:28:37.344385 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:37.344351 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-9b4747bc-kdqdg_f6768ac2-1740-48aa-9305-0b3884c1ebf2/istio-proxy/0.log" Oct 30 19:28:41.929929 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:41.929891 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pb8bs"] Oct 30 19:28:41.930472 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:41.930179 6960 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pb8bs" podUID="830b7d8c-c70c-4242-b81a-8be190df4aa1" containerName="istio-proxy" containerID="cri-o://8a884d0b39a6f7dbad9c234ccc97e1353f4933ead86878f61a364bb680e88c0b" gracePeriod=30 Oct 30 19:28:44.961000 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:44.960967 6960 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-855f9574cf-pb8bs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:28:44.961420 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:44.961023 6960 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pb8bs" podUID="830b7d8c-c70c-4242-b81a-8be190df4aa1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:28:47.090771 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:47.090734 6960 generic.go:356] "Generic (PLEG): container finished" podID="830b7d8c-c70c-4242-b81a-8be190df4aa1" containerID="8a884d0b39a6f7dbad9c234ccc97e1353f4933ead86878f61a364bb680e88c0b" exitCode=0 Oct 30 19:28:47.091195 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:47.090805 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pb8bs" event={"ID":"830b7d8c-c70c-4242-b81a-8be190df4aa1","Type":"ContainerDied","Data":"8a884d0b39a6f7dbad9c234ccc97e1353f4933ead86878f61a364bb680e88c0b"} Oct 30 19:28:47.180664 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:47.180642 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pb8bs" Oct 30 19:28:47.287700 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:47.287594 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/830b7d8c-c70c-4242-b81a-8be190df4aa1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/830b7d8c-c70c-4242-b81a-8be190df4aa1-istio-envoy\") pod \"830b7d8c-c70c-4242-b81a-8be190df4aa1\" (UID: \"830b7d8c-c70c-4242-b81a-8be190df4aa1\") " Oct 30 19:28:47.287700 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:47.287633 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/830b7d8c-c70c-4242-b81a-8be190df4aa1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/830b7d8c-c70c-4242-b81a-8be190df4aa1-istiod-ca-cert\") pod \"830b7d8c-c70c-4242-b81a-8be190df4aa1\" (UID: \"830b7d8c-c70c-4242-b81a-8be190df4aa1\") " Oct 30 19:28:47.287700 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:47.287684 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/830b7d8c-c70c-4242-b81a-8be190df4aa1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/830b7d8c-c70c-4242-b81a-8be190df4aa1-credential-socket\") pod \"830b7d8c-c70c-4242-b81a-8be190df4aa1\" (UID: \"830b7d8c-c70c-4242-b81a-8be190df4aa1\") " Oct 30 19:28:47.287992 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:47.287828 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/830b7d8c-c70c-4242-b81a-8be190df4aa1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/830b7d8c-c70c-4242-b81a-8be190df4aa1-workload-socket\") pod \"830b7d8c-c70c-4242-b81a-8be190df4aa1\" (UID: \"830b7d8c-c70c-4242-b81a-8be190df4aa1\") " Oct 30 19:28:47.287992 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:47.287861 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/830b7d8c-c70c-4242-b81a-8be190df4aa1-istio-token\" (UniqueName: \"kubernetes.io/projected/830b7d8c-c70c-4242-b81a-8be190df4aa1-istio-token\") pod \"830b7d8c-c70c-4242-b81a-8be190df4aa1\" (UID: \"830b7d8c-c70c-4242-b81a-8be190df4aa1\") " Oct 30 19:28:47.287992 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:47.287889 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/830b7d8c-c70c-4242-b81a-8be190df4aa1-kube-api-access-wgt44\" (UniqueName: \"kubernetes.io/projected/830b7d8c-c70c-4242-b81a-8be190df4aa1-kube-api-access-wgt44\") pod \"830b7d8c-c70c-4242-b81a-8be190df4aa1\" (UID: \"830b7d8c-c70c-4242-b81a-8be190df4aa1\") " Oct 30 19:28:47.288242 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:47.288047 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/830b7d8c-c70c-4242-b81a-8be190df4aa1-credential-socket" pod "830b7d8c-c70c-4242-b81a-8be190df4aa1" (UID: "830b7d8c-c70c-4242-b81a-8be190df4aa1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:28:47.288242 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:47.288112 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/830b7d8c-c70c-4242-b81a-8be190df4aa1-workload-socket" pod "830b7d8c-c70c-4242-b81a-8be190df4aa1" (UID: "830b7d8c-c70c-4242-b81a-8be190df4aa1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:28:47.288242 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:47.288174 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/830b7d8c-c70c-4242-b81a-8be190df4aa1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/830b7d8c-c70c-4242-b81a-8be190df4aa1-workload-certs\") pod \"830b7d8c-c70c-4242-b81a-8be190df4aa1\" (UID: \"830b7d8c-c70c-4242-b81a-8be190df4aa1\") " Oct 30 19:28:47.288242 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:47.288184 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/830b7d8c-c70c-4242-b81a-8be190df4aa1-istiod-ca-cert" pod "830b7d8c-c70c-4242-b81a-8be190df4aa1" (UID: "830b7d8c-c70c-4242-b81a-8be190df4aa1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:28:47.288750 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:47.288249 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/830b7d8c-c70c-4242-b81a-8be190df4aa1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/830b7d8c-c70c-4242-b81a-8be190df4aa1-istio-data\") pod \"830b7d8c-c70c-4242-b81a-8be190df4aa1\" (UID: \"830b7d8c-c70c-4242-b81a-8be190df4aa1\") " Oct 30 19:28:47.288750 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:47.288295 6960 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/830b7d8c-c70c-4242-b81a-8be190df4aa1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/830b7d8c-c70c-4242-b81a-8be190df4aa1-istio-podinfo\") pod \"830b7d8c-c70c-4242-b81a-8be190df4aa1\" (UID: \"830b7d8c-c70c-4242-b81a-8be190df4aa1\") " Oct 30 19:28:47.288750 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:47.288473 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/830b7d8c-c70c-4242-b81a-8be190df4aa1-istio-data" pod "830b7d8c-c70c-4242-b81a-8be190df4aa1" (UID: "830b7d8c-c70c-4242-b81a-8be190df4aa1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:28:47.288994 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:47.288806 6960 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/830b7d8c-c70c-4242-b81a-8be190df4aa1-credential-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:28:47.288994 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:47.288827 6960 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/830b7d8c-c70c-4242-b81a-8be190df4aa1-workload-socket\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:28:47.288994 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:47.288841 6960 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/830b7d8c-c70c-4242-b81a-8be190df4aa1-istio-data\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:28:47.288994 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:47.288855 6960 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/830b7d8c-c70c-4242-b81a-8be190df4aa1-istiod-ca-cert\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:28:47.289323 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:47.288406 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/830b7d8c-c70c-4242-b81a-8be190df4aa1-workload-certs" pod "830b7d8c-c70c-4242-b81a-8be190df4aa1" (UID: "830b7d8c-c70c-4242-b81a-8be190df4aa1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:28:47.290483 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:47.290438 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/830b7d8c-c70c-4242-b81a-8be190df4aa1-kube-api-access-wgt44" pod "830b7d8c-c70c-4242-b81a-8be190df4aa1" (UID: "830b7d8c-c70c-4242-b81a-8be190df4aa1"). InnerVolumeSpecName "kube-api-access-wgt44". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:28:47.290759 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:47.290736 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/830b7d8c-c70c-4242-b81a-8be190df4aa1-istio-envoy" pod "830b7d8c-c70c-4242-b81a-8be190df4aa1" (UID: "830b7d8c-c70c-4242-b81a-8be190df4aa1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:28:47.290850 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:47.290774 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/830b7d8c-c70c-4242-b81a-8be190df4aa1-istio-token" pod "830b7d8c-c70c-4242-b81a-8be190df4aa1" (UID: "830b7d8c-c70c-4242-b81a-8be190df4aa1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:28:47.292165 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:47.292129 6960 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/830b7d8c-c70c-4242-b81a-8be190df4aa1-istio-podinfo" pod "830b7d8c-c70c-4242-b81a-8be190df4aa1" (UID: "830b7d8c-c70c-4242-b81a-8be190df4aa1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:28:47.390225 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:47.390188 6960 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/830b7d8c-c70c-4242-b81a-8be190df4aa1-istio-token\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:28:47.390225 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:47.390220 6960 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgt44\" (UniqueName: \"kubernetes.io/projected/830b7d8c-c70c-4242-b81a-8be190df4aa1-kube-api-access-wgt44\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:28:47.390442 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:47.390234 6960 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/830b7d8c-c70c-4242-b81a-8be190df4aa1-workload-certs\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:28:47.390442 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:47.390246 6960 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/830b7d8c-c70c-4242-b81a-8be190df4aa1-istio-podinfo\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:28:47.390442 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:47.390259 6960 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/830b7d8c-c70c-4242-b81a-8be190df4aa1-istio-envoy\") on node \"ip-10-0-140-163.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:28:48.098010 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:48.097980 6960 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pb8bs" Oct 30 19:28:48.098460 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:48.097981 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pb8bs" event={"ID":"830b7d8c-c70c-4242-b81a-8be190df4aa1","Type":"ContainerDied","Data":"945789fdf6a8446db684fe5e517fcf94845a107e7757fa5bd6ed11cfb62e2db0"} Oct 30 19:28:48.098460 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:48.098135 6960 scope.go:122] "RemoveContainer" containerID="8a884d0b39a6f7dbad9c234ccc97e1353f4933ead86878f61a364bb680e88c0b" Oct 30 19:28:48.119964 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:48.119938 6960 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pb8bs"] Oct 30 19:28:48.124649 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:48.124626 6960 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-855f9574cf-pb8bs"] Oct 30 19:28:48.590901 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:48.590869 6960 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="830b7d8c-c70c-4242-b81a-8be190df4aa1" path="/var/lib/kubelet/pods/830b7d8c-c70c-4242-b81a-8be190df4aa1/volumes" Oct 30 19:28:50.832041 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:50.831998 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:51.044473 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:51.044440 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:53.931749 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:28:53.931706 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 19:28:55.058332 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:55.058297 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:28:55.149579 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:55.149551 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-zmdjg_e163025a-b19f-44e2-9058-b65f31bfbcc4/global-pull-secret-syncer/0.log" Oct 30 19:28:55.341907 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:55.341813 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" Oct 30 19:28:55.527336 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:28:55.527298 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-163.us-east-2.compute.internal_3b404058373133d538844156a64ac398/haproxy/0.log" Oct 30 19:29:03.134459 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:03.134421 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-hm48j_a624d24b-f689-4286-9602-c89d828a0e05/manager/0.log" Oct 30 19:29:03.199525 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:03.199498 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-f78tf_30c4e293-73c6-4d1f-8e76-6ffb15e03f51/kuadrant-console-plugin/0.log" Oct 30 19:29:05.502982 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:05.502953 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_13a85849-5c27-4ce8-87ee-059ae2765256/alertmanager/0.log" Oct 30 19:29:05.536925 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:05.536899 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_13a85849-5c27-4ce8-87ee-059ae2765256/config-reloader/0.log" Oct 30 19:29:05.569850 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:05.569829 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_13a85849-5c27-4ce8-87ee-059ae2765256/kube-rbac-proxy-web/0.log" Oct 30 19:29:05.599659 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:05.599636 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_13a85849-5c27-4ce8-87ee-059ae2765256/kube-rbac-proxy/0.log" Oct 30 19:29:05.629197 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:05.629172 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_13a85849-5c27-4ce8-87ee-059ae2765256/kube-rbac-proxy-metric/0.log" Oct 30 19:29:05.660779 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:05.660755 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_13a85849-5c27-4ce8-87ee-059ae2765256/prom-label-proxy/0.log" Oct 30 19:29:05.693929 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:05.693910 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_13a85849-5c27-4ce8-87ee-059ae2765256/init-config-reloader/0.log" Oct 30 19:29:05.791880 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:05.791812 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-7gk7g_4f8841c0-4cf3-45d9-9301-957d4fffe1ce/kube-state-metrics/0.log" Oct 30 19:29:05.815434 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:05.815408 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-7gk7g_4f8841c0-4cf3-45d9-9301-957d4fffe1ce/kube-rbac-proxy-main/0.log" Oct 30 19:29:05.848607 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:05.848585 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-7gk7g_4f8841c0-4cf3-45d9-9301-957d4fffe1ce/kube-rbac-proxy-self/0.log" Oct 30 19:29:05.902622 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:05.902600 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-b6d7bcd58-z2xvj_2069a85a-44e7-4c98-9f47-7fad4416c15b/metrics-server/0.log" Oct 30 19:29:05.936316 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:05.936294 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-r5d75_25377c15-6e29-46b3-9f1b-9b1fc38570f5/monitoring-plugin/0.log" Oct 30 19:29:06.076656 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:06.076631 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hh862_433ed181-b45e-45c0-a593-9cb0b6c09348/node-exporter/0.log" Oct 30 19:29:06.103309 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:06.103283 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hh862_433ed181-b45e-45c0-a593-9cb0b6c09348/kube-rbac-proxy/0.log" Oct 30 19:29:06.140236 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:06.140219 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hh862_433ed181-b45e-45c0-a593-9cb0b6c09348/init-textfile/0.log" Oct 30 19:29:06.308295 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:06.308268 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-8698b_312dae04-dfda-41da-90ca-36b3b44a12ea/kube-rbac-proxy-main/0.log" Oct 30 19:29:06.360205 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:06.360118 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-8698b_312dae04-dfda-41da-90ca-36b3b44a12ea/kube-rbac-proxy-self/0.log" Oct 30 19:29:06.391923 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:06.391895 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-8698b_312dae04-dfda-41da-90ca-36b3b44a12ea/openshift-state-metrics/0.log" Oct 30 19:29:06.444077 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:06.444055 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72/prometheus/0.log" Oct 30 19:29:06.483354 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:06.483330 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72/config-reloader/0.log" Oct 30 19:29:06.511797 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:06.511777 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72/thanos-sidecar/0.log" Oct 30 19:29:06.541206 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:06.541188 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72/kube-rbac-proxy-web/0.log" Oct 30 19:29:06.570771 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:06.570749 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72/kube-rbac-proxy/0.log" Oct 30 19:29:06.598420 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:06.598400 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72/kube-rbac-proxy-thanos/0.log" Oct 30 19:29:06.627126 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:06.627072 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fc1c29c1-a5b4-4ea2-8a3f-e757894b2a72/init-config-reloader/0.log" Oct 30 19:29:06.771286 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:06.771254 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-85d6484797-5gnbj_48000d54-fb98-4094-8160-b44be17f190c/telemeter-client/0.log" Oct 30 19:29:06.803030 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:06.803006 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-85d6484797-5gnbj_48000d54-fb98-4094-8160-b44be17f190c/reload/0.log" Oct 30 19:29:06.838075 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:06.838051 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-85d6484797-5gnbj_48000d54-fb98-4094-8160-b44be17f190c/kube-rbac-proxy/0.log" Oct 30 19:29:06.876445 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:06.876423 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c6f76d6d-k2h7h_158becf0-d6cf-4917-8329-4f64ae5a4ca4/thanos-query/0.log" Oct 30 19:29:06.913026 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:06.912958 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c6f76d6d-k2h7h_158becf0-d6cf-4917-8329-4f64ae5a4ca4/kube-rbac-proxy-web/0.log" Oct 30 19:29:06.944203 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:06.944181 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c6f76d6d-k2h7h_158becf0-d6cf-4917-8329-4f64ae5a4ca4/kube-rbac-proxy/0.log" Oct 30 19:29:06.980023 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:06.980001 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c6f76d6d-k2h7h_158becf0-d6cf-4917-8329-4f64ae5a4ca4/prom-label-proxy/0.log" Oct 30 19:29:07.011463 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.011439 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c6f76d6d-k2h7h_158becf0-d6cf-4917-8329-4f64ae5a4ca4/kube-rbac-proxy-rules/0.log" Oct 30 19:29:07.041062 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.041037 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c6f76d6d-k2h7h_158becf0-d6cf-4917-8329-4f64ae5a4ca4/kube-rbac-proxy-metrics/0.log" Oct 30 19:29:07.081904 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.081881 6960 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-nwzbq/perf-node-gather-daemonset-67w5q"] Oct 30 19:29:07.082280 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082268 6960 cpu_manager.go:414] "containerMap: removing container" podUID="3e8ff0f2-eaf3-434d-b877-f95964be8923" containerName="maas-api" Oct 30 19:29:07.082341 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082281 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e8ff0f2-eaf3-434d-b877-f95964be8923" containerName="maas-api" Oct 30 19:29:07.082341 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082296 6960 cpu_manager.go:414] "containerMap: removing container" podUID="ff4a8ee1-e818-4f1b-9b1e-f024237e3ec4" containerName="maas-api" Oct 30 19:29:07.082341 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082302 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff4a8ee1-e818-4f1b-9b1e-f024237e3ec4" containerName="maas-api" Oct 30 19:29:07.082341 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082311 6960 cpu_manager.go:414] "containerMap: removing container" podUID="da7fcb8d-db2d-4b43-a04f-22f4a9c0757b" containerName="maas-api" Oct 30 19:29:07.082341 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082316 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da7fcb8d-db2d-4b43-a04f-22f4a9c0757b" containerName="maas-api" Oct 30 19:29:07.082341 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082326 6960 cpu_manager.go:414] "containerMap: removing container" podUID="84ca40c5-7033-48c4-bcec-e9ac7c034a4e" containerName="main" Oct 30 19:29:07.082341 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082331 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84ca40c5-7033-48c4-bcec-e9ac7c034a4e" containerName="main" Oct 30 19:29:07.082341 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082337 6960 cpu_manager.go:414] "containerMap: removing container" podUID="d12464a9-d9c7-46e6-b3cc-ab0c16d75070" containerName="istio-proxy" Oct 30 19:29:07.082341 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082342 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d12464a9-d9c7-46e6-b3cc-ab0c16d75070" containerName="istio-proxy" Oct 30 19:29:07.082663 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082349 6960 cpu_manager.go:414] "containerMap: removing container" podUID="56a9fce0-2511-4217-afb8-feb2184fb974" containerName="maas-api" Oct 30 19:29:07.082663 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082354 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56a9fce0-2511-4217-afb8-feb2184fb974" containerName="maas-api" Oct 30 19:29:07.082663 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082360 6960 cpu_manager.go:414] "containerMap: removing container" podUID="8e4832f5-ee56-4ea9-82c2-9fffed013c35" containerName="sim" Oct 30 19:29:07.082663 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082364 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e4832f5-ee56-4ea9-82c2-9fffed013c35" containerName="sim" Oct 30 19:29:07.082663 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082371 6960 cpu_manager.go:414] "containerMap: removing container" podUID="e8701242-b59c-43b1-9c0a-011635edd419" containerName="main" Oct 30 19:29:07.082663 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082376 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8701242-b59c-43b1-9c0a-011635edd419" containerName="main" Oct 30 19:29:07.082663 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082381 6960 cpu_manager.go:414] "containerMap: removing container" podUID="0272be8c-cd91-4f7d-b183-c03a18fdadf5" containerName="payload-processing" Oct 30 19:29:07.082663 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082386 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0272be8c-cd91-4f7d-b183-c03a18fdadf5" containerName="payload-processing" Oct 30 19:29:07.082663 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082393 6960 cpu_manager.go:414] "containerMap: removing container" podUID="aeba7fc6-3fe2-4327-94cb-84c86ad4f1d1" containerName="maas-api" Oct 30 19:29:07.082663 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082397 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aeba7fc6-3fe2-4327-94cb-84c86ad4f1d1" containerName="maas-api" Oct 30 19:29:07.082663 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082404 6960 cpu_manager.go:414] "containerMap: removing container" podUID="696a841d-b511-4430-a236-d61ff1832626" containerName="maas-api" Oct 30 19:29:07.082663 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082409 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="696a841d-b511-4430-a236-d61ff1832626" containerName="maas-api" Oct 30 19:29:07.082663 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082415 6960 cpu_manager.go:414] "containerMap: removing container" podUID="cd525e12-1c0c-4ab5-91ac-9028a847cab6" containerName="maas-api" Oct 30 19:29:07.082663 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082419 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd525e12-1c0c-4ab5-91ac-9028a847cab6" containerName="maas-api" Oct 30 19:29:07.082663 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082425 6960 cpu_manager.go:414] "containerMap: removing container" podUID="c8690e67-62d5-4a1b-8632-a23a7d2f2843" containerName="payload-pre-processing" Oct 30 19:29:07.082663 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082430 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8690e67-62d5-4a1b-8632-a23a7d2f2843" containerName="payload-pre-processing" Oct 30 19:29:07.082663 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082437 6960 cpu_manager.go:414] "containerMap: removing container" podUID="ccb27153-3ebf-4eaa-80a1-e460de5d78d0" containerName="payload-pre-processing" Oct 30 19:29:07.082663 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082441 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccb27153-3ebf-4eaa-80a1-e460de5d78d0" containerName="payload-pre-processing" Oct 30 19:29:07.082663 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082450 6960 cpu_manager.go:414] "containerMap: removing container" podUID="efd98aa3-a350-4bf9-82ec-34f7b7917adc" containerName="maas-api" Oct 30 19:29:07.082663 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082455 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efd98aa3-a350-4bf9-82ec-34f7b7917adc" containerName="maas-api" Oct 30 19:29:07.082663 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082462 6960 cpu_manager.go:414] "containerMap: removing container" podUID="b5a06a95-d251-46b6-889c-7e7d5e2ca925" containerName="payload-processing" Oct 30 19:29:07.082663 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082466 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5a06a95-d251-46b6-889c-7e7d5e2ca925" containerName="payload-processing" Oct 30 19:29:07.082663 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082473 6960 cpu_manager.go:414] "containerMap: removing container" podUID="1ff59ea2-88a1-4532-a214-6bafc573b44f" containerName="payload-pre-processing" Oct 30 19:29:07.082663 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082478 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ff59ea2-88a1-4532-a214-6bafc573b44f" containerName="payload-pre-processing" Oct 30 19:29:07.082663 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082486 6960 cpu_manager.go:414] "containerMap: removing container" podUID="5dc3436e-00d1-4ac1-9ea4-feaa1d92a15f" containerName="maas-api" Oct 30 19:29:07.082663 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082490 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5dc3436e-00d1-4ac1-9ea4-feaa1d92a15f" containerName="maas-api" Oct 30 19:29:07.082663 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082496 6960 cpu_manager.go:414] "containerMap: removing container" podUID="21b44f0f-4b22-4369-9045-c9065515f816" containerName="payload-processing" Oct 30 19:29:07.082663 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082501 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21b44f0f-4b22-4369-9045-c9065515f816" containerName="payload-processing" Oct 30 19:29:07.082663 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082508 6960 cpu_manager.go:414] "containerMap: removing container" podUID="f51fd746-cd85-4608-96a6-f54afa0ec34d" containerName="main" Oct 30 19:29:07.082663 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082512 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f51fd746-cd85-4608-96a6-f54afa0ec34d" containerName="main" Oct 30 19:29:07.082663 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082518 6960 cpu_manager.go:414] "containerMap: removing container" podUID="aef219d1-2021-4cea-b80b-ed995446c87c" containerName="maas-api" Oct 30 19:29:07.082663 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082522 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aef219d1-2021-4cea-b80b-ed995446c87c" containerName="maas-api" Oct 30 19:29:07.082663 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082528 6960 cpu_manager.go:414] "containerMap: removing container" podUID="58c23db7-11cb-4ce2-a67e-fe698e89c71d" containerName="main" Oct 30 19:29:07.082663 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082532 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58c23db7-11cb-4ce2-a67e-fe698e89c71d" containerName="main" Oct 30 19:29:07.082663 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082538 6960 cpu_manager.go:414] "containerMap: removing container" podUID="eb97b10a-d1c7-40a0-b182-2377facd98bc" containerName="istio-proxy" Oct 30 19:29:07.082663 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082543 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb97b10a-d1c7-40a0-b182-2377facd98bc" containerName="istio-proxy" Oct 30 19:29:07.082663 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082549 6960 cpu_manager.go:414] "containerMap: removing container" podUID="a59257f8-0671-461d-ac61-4956961be8f8" containerName="maas-api" Oct 30 19:29:07.082663 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082553 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a59257f8-0671-461d-ac61-4956961be8f8" containerName="maas-api" Oct 30 19:29:07.082663 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082561 6960 cpu_manager.go:414] "containerMap: removing container" podUID="b06be103-9039-4229-bb04-d464932e988f" containerName="istio-proxy" Oct 30 19:29:07.082663 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082565 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b06be103-9039-4229-bb04-d464932e988f" containerName="istio-proxy" Oct 30 19:29:07.082663 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082572 6960 cpu_manager.go:414] "containerMap: removing container" podUID="9c25987f-3a86-4c2e-ae7c-41080e0a3e24" containerName="discovery" Oct 30 19:29:07.082663 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082576 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c25987f-3a86-4c2e-ae7c-41080e0a3e24" containerName="discovery" Oct 30 19:29:07.082663 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082583 6960 cpu_manager.go:414] "containerMap: removing container" podUID="ce70492e-c727-4dc1-b2ee-bf278d29a45b" containerName="main" Oct 30 19:29:07.082663 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082587 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce70492e-c727-4dc1-b2ee-bf278d29a45b" containerName="main" Oct 30 19:29:07.082663 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082593 6960 cpu_manager.go:414] "containerMap: removing container" podUID="71c3b467-e87a-40eb-ad20-7175b0db9e49" containerName="istio-proxy" Oct 30 19:29:07.082663 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082597 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71c3b467-e87a-40eb-ad20-7175b0db9e49" containerName="istio-proxy" Oct 30 19:29:07.082663 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082602 6960 cpu_manager.go:414] "containerMap: removing container" podUID="904f96ca-921a-4d2e-8245-46500d6b7d0a" containerName="istio-proxy" Oct 30 19:29:07.082663 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082606 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="904f96ca-921a-4d2e-8245-46500d6b7d0a" containerName="istio-proxy" Oct 30 19:29:07.082663 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082612 6960 cpu_manager.go:414] "containerMap: removing container" podUID="755750a6-9b02-43c7-adfd-00f2073020fe" containerName="istio-proxy" Oct 30 19:29:07.082663 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082617 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="755750a6-9b02-43c7-adfd-00f2073020fe" containerName="istio-proxy" Oct 30 19:29:07.082663 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082627 6960 cpu_manager.go:414] "containerMap: removing container" podUID="67bc209e-b148-4464-912e-f1ba5f372c50" containerName="istio-proxy" Oct 30 19:29:07.082663 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082631 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67bc209e-b148-4464-912e-f1ba5f372c50" containerName="istio-proxy" Oct 30 19:29:07.082663 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082639 6960 cpu_manager.go:414] "containerMap: removing container" podUID="e5fd8f33-8662-4f90-9817-e601f694bf1d" containerName="main" Oct 30 19:29:07.082663 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082643 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5fd8f33-8662-4f90-9817-e601f694bf1d" containerName="main" Oct 30 19:29:07.082663 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082651 6960 cpu_manager.go:414] "containerMap: removing container" podUID="943f7840-8d54-4845-b7ec-c0da72ba9c67" containerName="main" Oct 30 19:29:07.082663 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082655 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="943f7840-8d54-4845-b7ec-c0da72ba9c67" containerName="main" Oct 30 19:29:07.082663 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082661 6960 cpu_manager.go:414] "containerMap: removing container" podUID="6804806b-3a5b-425d-9ae7-55bda5f68a4e" containerName="payload-processing" Oct 30 19:29:07.082663 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082665 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6804806b-3a5b-425d-9ae7-55bda5f68a4e" containerName="payload-processing" Oct 30 19:29:07.082663 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082672 6960 cpu_manager.go:414] "containerMap: removing container" podUID="68ec602e-60f3-4540-bfcf-9b29cdcfda2f" containerName="maas-api" Oct 30 19:29:07.082663 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082677 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68ec602e-60f3-4540-bfcf-9b29cdcfda2f" containerName="maas-api" Oct 30 19:29:07.082663 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082683 6960 cpu_manager.go:414] "containerMap: removing container" podUID="94b983c1-08fc-4f62-85de-e24696834c7c" containerName="istio-proxy" Oct 30 19:29:07.082663 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082688 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94b983c1-08fc-4f62-85de-e24696834c7c" containerName="istio-proxy" Oct 30 19:29:07.085098 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082695 6960 cpu_manager.go:414] "containerMap: removing container" podUID="6ebea224-1a00-496f-a41e-1d7f2a294a5b" containerName="payload-pre-processing" Oct 30 19:29:07.085098 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082700 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ebea224-1a00-496f-a41e-1d7f2a294a5b" containerName="payload-pre-processing" Oct 30 19:29:07.085098 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082706 6960 cpu_manager.go:414] "containerMap: removing container" podUID="4522747f-f5bf-481e-a35c-58e3475a3221" containerName="istio-proxy" Oct 30 19:29:07.085098 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082710 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4522747f-f5bf-481e-a35c-58e3475a3221" containerName="istio-proxy" Oct 30 19:29:07.085098 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082718 6960 cpu_manager.go:414] "containerMap: removing container" podUID="f1ef57ac-d33d-4998-b22b-87b2aed6ba8f" containerName="istio-proxy" Oct 30 19:29:07.085098 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082722 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1ef57ac-d33d-4998-b22b-87b2aed6ba8f" containerName="istio-proxy" Oct 30 19:29:07.085098 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082729 6960 cpu_manager.go:414] "containerMap: removing container" podUID="7b416cfb-588b-4dde-bd77-9d5fca225a87" containerName="istio-proxy" Oct 30 19:29:07.085098 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082733 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b416cfb-588b-4dde-bd77-9d5fca225a87" containerName="istio-proxy" Oct 30 19:29:07.085098 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082739 6960 cpu_manager.go:414] "containerMap: removing container" podUID="0ac809bd-4083-4519-b7b5-903a22885985" containerName="maas-api" Oct 30 19:29:07.085098 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082743 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ac809bd-4083-4519-b7b5-903a22885985" containerName="maas-api" Oct 30 19:29:07.085098 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082752 6960 cpu_manager.go:414] "containerMap: removing container" podUID="cc64ebb9-44e9-435a-9db0-d53ce5f139ff" containerName="main" Oct 30 19:29:07.085098 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082756 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc64ebb9-44e9-435a-9db0-d53ce5f139ff" containerName="main" Oct 30 19:29:07.085098 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082764 6960 cpu_manager.go:414] "containerMap: removing container" podUID="c610dae0-c986-4c71-b430-9e1e7bd73e9b" containerName="maas-api" Oct 30 19:29:07.085098 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082769 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c610dae0-c986-4c71-b430-9e1e7bd73e9b" containerName="maas-api" Oct 30 19:29:07.085098 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082774 6960 cpu_manager.go:414] "containerMap: removing container" podUID="afe56fb6-3076-47f1-92c7-a45b0426bdad" containerName="istio-proxy" Oct 30 19:29:07.085098 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082779 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afe56fb6-3076-47f1-92c7-a45b0426bdad" containerName="istio-proxy" Oct 30 19:29:07.085098 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082785 6960 cpu_manager.go:414] "containerMap: removing container" podUID="f2fbe8d6-d7b0-4814-a665-e47be1a45176" containerName="istio-proxy" Oct 30 19:29:07.085098 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082789 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2fbe8d6-d7b0-4814-a665-e47be1a45176" containerName="istio-proxy" Oct 30 19:29:07.085098 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082795 6960 cpu_manager.go:414] "containerMap: removing container" podUID="830b7d8c-c70c-4242-b81a-8be190df4aa1" containerName="istio-proxy" Oct 30 19:29:07.085098 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082799 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="830b7d8c-c70c-4242-b81a-8be190df4aa1" containerName="istio-proxy" Oct 30 19:29:07.085098 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082804 6960 cpu_manager.go:414] "containerMap: removing container" podUID="7998c864-d4ca-4b59-9ef6-0838726a0c8c" containerName="maas-api" Oct 30 19:29:07.085098 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082809 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7998c864-d4ca-4b59-9ef6-0838726a0c8c" containerName="maas-api" Oct 30 19:29:07.085098 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082814 6960 cpu_manager.go:414] "containerMap: removing container" podUID="48c4bb82-3969-4e2d-a303-ef614827b953" containerName="maas-api" Oct 30 19:29:07.085098 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082818 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48c4bb82-3969-4e2d-a303-ef614827b953" containerName="maas-api" Oct 30 19:29:07.085098 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082823 6960 cpu_manager.go:414] "containerMap: removing container" podUID="f3541c21-2190-4ba9-a2aa-286582db6f34" containerName="payload-processing" Oct 30 19:29:07.085098 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082827 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3541c21-2190-4ba9-a2aa-286582db6f34" containerName="payload-processing" Oct 30 19:29:07.085098 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082840 6960 cpu_manager.go:414] "containerMap: removing container" podUID="50c12729-1387-436a-bee0-9eb3316db708" containerName="istio-proxy" Oct 30 19:29:07.085098 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082845 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50c12729-1387-436a-bee0-9eb3316db708" containerName="istio-proxy" Oct 30 19:29:07.085098 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082852 6960 cpu_manager.go:414] "containerMap: removing container" podUID="67a3f38a-10e8-4826-925c-520ce8297f11" containerName="payload-processing" Oct 30 19:29:07.085098 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082856 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67a3f38a-10e8-4826-925c-520ce8297f11" containerName="payload-processing" Oct 30 19:29:07.085098 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082863 6960 cpu_manager.go:414] "containerMap: removing container" podUID="da02a9d7-7b75-401c-9134-8e45d7303175" containerName="istio-proxy" Oct 30 19:29:07.085098 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082867 6960 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da02a9d7-7b75-401c-9134-8e45d7303175" containerName="istio-proxy" Oct 30 19:29:07.085098 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082930 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="67a3f38a-10e8-4826-925c-520ce8297f11" containerName="payload-processing" Oct 30 19:29:07.085098 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082940 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="da02a9d7-7b75-401c-9134-8e45d7303175" containerName="istio-proxy" Oct 30 19:29:07.085098 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082945 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e8ff0f2-eaf3-434d-b877-f95964be8923" containerName="maas-api" Oct 30 19:29:07.085098 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082953 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff4a8ee1-e818-4f1b-9b1e-f024237e3ec4" containerName="maas-api" Oct 30 19:29:07.085098 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082957 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="da7fcb8d-db2d-4b43-a04f-22f4a9c0757b" containerName="maas-api" Oct 30 19:29:07.085098 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082964 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="84ca40c5-7033-48c4-bcec-e9ac7c034a4e" containerName="main" Oct 30 19:29:07.085098 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082970 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="d12464a9-d9c7-46e6-b3cc-ab0c16d75070" containerName="istio-proxy" Oct 30 19:29:07.085098 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082976 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="56a9fce0-2511-4217-afb8-feb2184fb974" containerName="maas-api" Oct 30 19:29:07.085098 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082980 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e4832f5-ee56-4ea9-82c2-9fffed013c35" containerName="sim" Oct 30 19:29:07.085098 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082985 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="0272be8c-cd91-4f7d-b183-c03a18fdadf5" containerName="payload-processing" Oct 30 19:29:07.085098 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082989 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8701242-b59c-43b1-9c0a-011635edd419" containerName="main" Oct 30 19:29:07.085098 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.082995 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="aeba7fc6-3fe2-4327-94cb-84c86ad4f1d1" containerName="maas-api" Oct 30 19:29:07.085098 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.083001 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="696a841d-b511-4430-a236-d61ff1832626" containerName="maas-api" Oct 30 19:29:07.085098 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.083005 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd525e12-1c0c-4ab5-91ac-9028a847cab6" containerName="maas-api" Oct 30 19:29:07.085098 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.083011 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8690e67-62d5-4a1b-8632-a23a7d2f2843" containerName="payload-pre-processing" Oct 30 19:29:07.085098 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.083016 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccb27153-3ebf-4eaa-80a1-e460de5d78d0" containerName="payload-pre-processing" Oct 30 19:29:07.085098 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.083022 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="efd98aa3-a350-4bf9-82ec-34f7b7917adc" containerName="maas-api" Oct 30 19:29:07.085098 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.083028 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5a06a95-d251-46b6-889c-7e7d5e2ca925" containerName="payload-processing" Oct 30 19:29:07.085098 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.083032 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ff59ea2-88a1-4532-a214-6bafc573b44f" containerName="payload-pre-processing" Oct 30 19:29:07.085098 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.083037 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="5dc3436e-00d1-4ac1-9ea4-feaa1d92a15f" containerName="maas-api" Oct 30 19:29:07.085098 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.083041 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="21b44f0f-4b22-4369-9045-c9065515f816" containerName="payload-processing" Oct 30 19:29:07.085098 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.083046 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="f51fd746-cd85-4608-96a6-f54afa0ec34d" containerName="main" Oct 30 19:29:07.085098 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.083050 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="aef219d1-2021-4cea-b80b-ed995446c87c" containerName="maas-api" Oct 30 19:29:07.085098 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.083055 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="58c23db7-11cb-4ce2-a67e-fe698e89c71d" containerName="main" Oct 30 19:29:07.085098 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.083059 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb97b10a-d1c7-40a0-b182-2377facd98bc" containerName="istio-proxy" Oct 30 19:29:07.085098 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.083065 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="a59257f8-0671-461d-ac61-4956961be8f8" containerName="maas-api" Oct 30 19:29:07.085098 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.083069 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="b06be103-9039-4229-bb04-d464932e988f" containerName="istio-proxy" Oct 30 19:29:07.085098 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.083074 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c25987f-3a86-4c2e-ae7c-41080e0a3e24" containerName="discovery" Oct 30 19:29:07.085098 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.083078 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce70492e-c727-4dc1-b2ee-bf278d29a45b" containerName="main" Oct 30 19:29:07.085098 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.083082 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="71c3b467-e87a-40eb-ad20-7175b0db9e49" containerName="istio-proxy" Oct 30 19:29:07.085098 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.083087 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="904f96ca-921a-4d2e-8245-46500d6b7d0a" containerName="istio-proxy" Oct 30 19:29:07.085098 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.083091 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="755750a6-9b02-43c7-adfd-00f2073020fe" containerName="istio-proxy" Oct 30 19:29:07.085098 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.083097 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="67bc209e-b148-4464-912e-f1ba5f372c50" containerName="istio-proxy" Oct 30 19:29:07.085098 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.083102 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5fd8f33-8662-4f90-9817-e601f694bf1d" containerName="main" Oct 30 19:29:07.085098 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.083106 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="943f7840-8d54-4845-b7ec-c0da72ba9c67" containerName="main" Oct 30 19:29:07.085098 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.083110 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="6804806b-3a5b-425d-9ae7-55bda5f68a4e" containerName="payload-processing" Oct 30 19:29:07.085098 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.083115 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="68ec602e-60f3-4540-bfcf-9b29cdcfda2f" containerName="maas-api" Oct 30 19:29:07.085098 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.083120 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="94b983c1-08fc-4f62-85de-e24696834c7c" containerName="istio-proxy" Oct 30 19:29:07.085098 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.083125 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ebea224-1a00-496f-a41e-1d7f2a294a5b" containerName="payload-pre-processing" Oct 30 19:29:07.085098 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.083130 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="4522747f-f5bf-481e-a35c-58e3475a3221" containerName="istio-proxy" Oct 30 19:29:07.085098 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.083136 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1ef57ac-d33d-4998-b22b-87b2aed6ba8f" containerName="istio-proxy" Oct 30 19:29:07.085098 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.083154 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b416cfb-588b-4dde-bd77-9d5fca225a87" containerName="istio-proxy" Oct 30 19:29:07.085098 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.083160 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ac809bd-4083-4519-b7b5-903a22885985" containerName="maas-api" Oct 30 19:29:07.085098 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.083165 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc64ebb9-44e9-435a-9db0-d53ce5f139ff" containerName="main" Oct 30 19:29:07.085098 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.083170 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="c610dae0-c986-4c71-b430-9e1e7bd73e9b" containerName="maas-api" Oct 30 19:29:07.085098 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.083174 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="afe56fb6-3076-47f1-92c7-a45b0426bdad" containerName="istio-proxy" Oct 30 19:29:07.085098 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.083179 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2fbe8d6-d7b0-4814-a665-e47be1a45176" containerName="istio-proxy" Oct 30 19:29:07.085098 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.083184 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="830b7d8c-c70c-4242-b81a-8be190df4aa1" containerName="istio-proxy" Oct 30 19:29:07.085098 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.083189 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="7998c864-d4ca-4b59-9ef6-0838726a0c8c" containerName="maas-api" Oct 30 19:29:07.085098 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.083194 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="48c4bb82-3969-4e2d-a303-ef614827b953" containerName="maas-api" Oct 30 19:29:07.085098 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.083199 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3541c21-2190-4ba9-a2aa-286582db6f34" containerName="payload-processing" Oct 30 19:29:07.085098 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.083203 6960 memory_manager.go:365] "RemoveStaleState removing state" podUID="50c12729-1387-436a-bee0-9eb3316db708" containerName="istio-proxy" Oct 30 19:29:07.087974 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.086232 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nwzbq/perf-node-gather-daemonset-67w5q" Oct 30 19:29:07.094408 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.094383 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nwzbq/perf-node-gather-daemonset-67w5q"] Oct 30 19:29:07.174356 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.174289 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9dd24528-fbe3-4716-83d0-fcf71389f9a6-podres\") pod \"perf-node-gather-daemonset-67w5q\" (UID: \"9dd24528-fbe3-4716-83d0-fcf71389f9a6\") " pod="openshift-must-gather-nwzbq/perf-node-gather-daemonset-67w5q" Oct 30 19:29:07.174356 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.174350 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9dd24528-fbe3-4716-83d0-fcf71389f9a6-lib-modules\") pod \"perf-node-gather-daemonset-67w5q\" (UID: \"9dd24528-fbe3-4716-83d0-fcf71389f9a6\") " pod="openshift-must-gather-nwzbq/perf-node-gather-daemonset-67w5q" Oct 30 19:29:07.174521 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.174443 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gghqx\" (UniqueName: \"kubernetes.io/projected/9dd24528-fbe3-4716-83d0-fcf71389f9a6-kube-api-access-gghqx\") pod \"perf-node-gather-daemonset-67w5q\" (UID: \"9dd24528-fbe3-4716-83d0-fcf71389f9a6\") " pod="openshift-must-gather-nwzbq/perf-node-gather-daemonset-67w5q" Oct 30 19:29:07.174560 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.174544 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9dd24528-fbe3-4716-83d0-fcf71389f9a6-sys\") pod \"perf-node-gather-daemonset-67w5q\" (UID: \"9dd24528-fbe3-4716-83d0-fcf71389f9a6\") " pod="openshift-must-gather-nwzbq/perf-node-gather-daemonset-67w5q" Oct 30 19:29:07.174597 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.174579 6960 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9dd24528-fbe3-4716-83d0-fcf71389f9a6-proc\") pod \"perf-node-gather-daemonset-67w5q\" (UID: \"9dd24528-fbe3-4716-83d0-fcf71389f9a6\") " pod="openshift-must-gather-nwzbq/perf-node-gather-daemonset-67w5q" Oct 30 19:29:07.183947 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.183928 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nwzbq\"/\"kube-root-ca.crt\"" Oct 30 19:29:07.184153 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.184129 6960 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nwzbq\"/\"openshift-service-ca.crt\"" Oct 30 19:29:07.185321 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.185304 6960 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-nwzbq\"/\"default-dockercfg-jgvfw\"" Oct 30 19:29:07.275512 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.275483 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9dd24528-fbe3-4716-83d0-fcf71389f9a6-podres\") pod \"perf-node-gather-daemonset-67w5q\" (UID: \"9dd24528-fbe3-4716-83d0-fcf71389f9a6\") " pod="openshift-must-gather-nwzbq/perf-node-gather-daemonset-67w5q" Oct 30 19:29:07.275666 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.275525 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9dd24528-fbe3-4716-83d0-fcf71389f9a6-lib-modules\") pod \"perf-node-gather-daemonset-67w5q\" (UID: \"9dd24528-fbe3-4716-83d0-fcf71389f9a6\") " pod="openshift-must-gather-nwzbq/perf-node-gather-daemonset-67w5q" Oct 30 19:29:07.275666 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.275549 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gghqx\" (UniqueName: \"kubernetes.io/projected/9dd24528-fbe3-4716-83d0-fcf71389f9a6-kube-api-access-gghqx\") pod \"perf-node-gather-daemonset-67w5q\" (UID: \"9dd24528-fbe3-4716-83d0-fcf71389f9a6\") " pod="openshift-must-gather-nwzbq/perf-node-gather-daemonset-67w5q" Oct 30 19:29:07.275666 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.275593 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9dd24528-fbe3-4716-83d0-fcf71389f9a6-sys\") pod \"perf-node-gather-daemonset-67w5q\" (UID: \"9dd24528-fbe3-4716-83d0-fcf71389f9a6\") " pod="openshift-must-gather-nwzbq/perf-node-gather-daemonset-67w5q" Oct 30 19:29:07.275666 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.275615 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9dd24528-fbe3-4716-83d0-fcf71389f9a6-podres\") pod \"perf-node-gather-daemonset-67w5q\" (UID: \"9dd24528-fbe3-4716-83d0-fcf71389f9a6\") " pod="openshift-must-gather-nwzbq/perf-node-gather-daemonset-67w5q" Oct 30 19:29:07.275666 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.275625 6960 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9dd24528-fbe3-4716-83d0-fcf71389f9a6-proc\") pod \"perf-node-gather-daemonset-67w5q\" (UID: \"9dd24528-fbe3-4716-83d0-fcf71389f9a6\") " pod="openshift-must-gather-nwzbq/perf-node-gather-daemonset-67w5q" Oct 30 19:29:07.275875 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.275667 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9dd24528-fbe3-4716-83d0-fcf71389f9a6-proc\") pod \"perf-node-gather-daemonset-67w5q\" (UID: \"9dd24528-fbe3-4716-83d0-fcf71389f9a6\") " pod="openshift-must-gather-nwzbq/perf-node-gather-daemonset-67w5q" Oct 30 19:29:07.275875 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.275701 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9dd24528-fbe3-4716-83d0-fcf71389f9a6-sys\") pod \"perf-node-gather-daemonset-67w5q\" (UID: \"9dd24528-fbe3-4716-83d0-fcf71389f9a6\") " pod="openshift-must-gather-nwzbq/perf-node-gather-daemonset-67w5q" Oct 30 19:29:07.275875 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.275641 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9dd24528-fbe3-4716-83d0-fcf71389f9a6-lib-modules\") pod \"perf-node-gather-daemonset-67w5q\" (UID: \"9dd24528-fbe3-4716-83d0-fcf71389f9a6\") " pod="openshift-must-gather-nwzbq/perf-node-gather-daemonset-67w5q" Oct 30 19:29:07.287690 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.287668 6960 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gghqx\" (UniqueName: \"kubernetes.io/projected/9dd24528-fbe3-4716-83d0-fcf71389f9a6-kube-api-access-gghqx\") pod \"perf-node-gather-daemonset-67w5q\" (UID: \"9dd24528-fbe3-4716-83d0-fcf71389f9a6\") " pod="openshift-must-gather-nwzbq/perf-node-gather-daemonset-67w5q" Oct 30 19:29:07.399677 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.399648 6960 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nwzbq/perf-node-gather-daemonset-67w5q" Oct 30 19:29:07.670995 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:07.670973 6960 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nwzbq/perf-node-gather-daemonset-67w5q"] Oct 30 19:29:07.672999 ip-10-0-140-163 kubenswrapper[6960]: W1030 19:29:07.672962 6960 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod9dd24528_fbe3_4716_83d0_fcf71389f9a6.slice/crio-6a1af5eab825982180e48224f343c7bfc38e2ec1dd0f55e4dd91223c3eb20d72 WatchSource:0}: Error finding container 6a1af5eab825982180e48224f343c7bfc38e2ec1dd0f55e4dd91223c3eb20d72: Status 404 returned error can't find the container with id 6a1af5eab825982180e48224f343c7bfc38e2ec1dd0f55e4dd91223c3eb20d72 Oct 30 19:29:08.198770 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:08.198738 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nwzbq/perf-node-gather-daemonset-67w5q" event={"ID":"9dd24528-fbe3-4716-83d0-fcf71389f9a6","Type":"ContainerStarted","Data":"380777d512ef638d6db0d9da206166afdaf88f894c222ba6019a22a19edda177"} Oct 30 19:29:08.198954 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:08.198793 6960 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nwzbq/perf-node-gather-daemonset-67w5q" event={"ID":"9dd24528-fbe3-4716-83d0-fcf71389f9a6","Type":"ContainerStarted","Data":"6a1af5eab825982180e48224f343c7bfc38e2ec1dd0f55e4dd91223c3eb20d72"} Oct 30 19:29:08.198954 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:08.198824 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-nwzbq/perf-node-gather-daemonset-67w5q" Oct 30 19:29:08.217048 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:08.216997 6960 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-nwzbq/perf-node-gather-daemonset-67w5q" podStartSLOduration=1.216981394 podStartE2EDuration="1.216981394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:29:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:08.216309559 +0000 UTC m=+3315.745340696" watchObservedRunningTime="2026-10-30 19:29:08.216981394 +0000 UTC m=+3315.746012535" Oct 30 19:29:09.534384 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:09.534353 6960 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4g8sn_b4961ad6-2fc3-4c01-a565-6f66068d2958/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:12.257794 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:12.257754 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-xkxkl_13296062-8ceb-4447-9418-d52de1a4b712/manager/0.log" Oct 30 19:29:13.099157 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:13.099114 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-69dbdc8f84-kt9h9_7d18009e-009c-40be-9143-e468d1bdfc41/manager/0.log" Oct 30 19:29:13.114458 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:13.114430 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-spxts_d99c9271-c3ea-4f9b-8668-c0348c766af0/server/0.log" Oct 30 19:29:13.217265 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:13.217231 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vfn4g_6c1405e1-dd94-4151-9b4f-6392155a2023/dns/0.log" Oct 30 19:29:13.257241 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:13.257207 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vfn4g_6c1405e1-dd94-4151-9b4f-6392155a2023/kube-rbac-proxy/0.log" Oct 30 19:29:13.295031 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:13.295005 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bk5br_05cad8b6-56f5-4bd1-b12d-b2ae456b7027/dns-node-resolver/0.log" Oct 30 19:29:13.460366 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:13.460257 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-xkxkl_13296062-8ceb-4447-9418-d52de1a4b712/manager/0.log" Oct 30 19:29:13.842731 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:13.842697 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-k5ljv_5b43299a-0f21-4d69-8167-6f8739c40a9b/proxy/0.log" Oct 30 19:29:14.103424 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:14.103322 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-69dbdc8f84-kt9h9_7d18009e-009c-40be-9143-e468d1bdfc41/manager/0.log" Oct 30 19:29:14.117194 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:14.117137 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-spxts_d99c9271-c3ea-4f9b-8668-c0348c766af0/server/0.log" Oct 30 19:29:14.215666 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:14.215630 6960 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-nwzbq/perf-node-gather-daemonset-67w5q" Oct 30 19:29:14.360100 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:14.360014 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qrdc6_b0f929fc-a9ea-4935-9fe7-770c5e7d4de5/node-ca/0.log" Oct 30 19:29:14.824689 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:14.824661 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-k5ljv_5b43299a-0f21-4d69-8167-6f8739c40a9b/proxy/0.log" Oct 30 19:29:15.919123 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:15.919069 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68d996c55722t75_9b0ef8f8-8b08-40e1-8338-fd48c63cfcf7/istio-proxy/0.log" Oct 30 19:29:16.189478 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:16.189389 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-xkxkl_13296062-8ceb-4447-9418-d52de1a4b712/manager/0.log" Oct 30 19:29:17.010928 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:17.010889 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-69dbdc8f84-kt9h9_7d18009e-009c-40be-9143-e468d1bdfc41/manager/0.log" Oct 30 19:29:17.068192 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:17.068135 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-spxts_d99c9271-c3ea-4f9b-8668-c0348c766af0/server/0.log" Oct 30 19:29:17.522962 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:17.522925 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-xkxkl_13296062-8ceb-4447-9418-d52de1a4b712/manager/0.log" Oct 30 19:29:17.827714 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:17.827681 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-65c8c79996-jjcwx_fa35d05a-b5c7-40b2-ac7a-6d72ebf85a69/kube-auth-proxy/0.log" Oct 30 19:29:17.938210 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:17.938182 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-k5ljv_5b43299a-0f21-4d69-8167-6f8739c40a9b/proxy/0.log" Oct 30 19:29:17.941123 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:17.941088 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-dslrt_93b3f3a7-cd85-4996-88fe-7762155b3241/istio-proxy/0.log" Oct 30 19:29:18.127282 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:18.127198 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-9b4747bc-kdqdg_f6768ac2-1740-48aa-9305-0b3884c1ebf2/istio-proxy/0.log" Oct 30 19:29:18.192829 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:18.192798 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-69dbdc8f84-kt9h9_7d18009e-009c-40be-9143-e468d1bdfc41/manager/0.log" Oct 30 19:29:18.221999 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:18.221975 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-spxts_d99c9271-c3ea-4f9b-8668-c0348c766af0/server/0.log" Oct 30 19:29:19.046742 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:19.046709 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-k5ljv_5b43299a-0f21-4d69-8167-6f8739c40a9b/proxy/0.log" Oct 30 19:29:20.326051 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:20.326017 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-xkxkl_13296062-8ceb-4447-9418-d52de1a4b712/manager/0.log" Oct 30 19:29:20.400847 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:20.400816 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68d996c55722t75_9b0ef8f8-8b08-40e1-8338-fd48c63cfcf7/istio-proxy/0.log" Oct 30 19:29:21.104116 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:21.104081 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-69dbdc8f84-kt9h9_7d18009e-009c-40be-9143-e468d1bdfc41/manager/0.log" Oct 30 19:29:21.150182 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:21.150125 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-spxts_d99c9271-c3ea-4f9b-8668-c0348c766af0/server/0.log" Oct 30 19:29:21.182210 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:21.182175 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bz2mv_91716cdd-99fb-4140-a52c-6d357ac74fb1/serve-healthcheck-canary/0.log" Oct 30 19:29:21.734918 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:21.734840 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-65c8c79996-jjcwx_fa35d05a-b5c7-40b2-ac7a-6d72ebf85a69/kube-auth-proxy/0.log" Oct 30 19:29:21.737036 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:21.737015 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-xkxkl_13296062-8ceb-4447-9418-d52de1a4b712/manager/0.log" Oct 30 19:29:21.805949 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:21.805909 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-dslrt_93b3f3a7-cd85-4996-88fe-7762155b3241/istio-proxy/0.log" Oct 30 19:29:21.977626 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:21.977592 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-9b4747bc-kdqdg_f6768ac2-1740-48aa-9305-0b3884c1ebf2/istio-proxy/0.log" Oct 30 19:29:22.090203 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:22.090170 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-k5ljv_5b43299a-0f21-4d69-8167-6f8739c40a9b/proxy/0.log" Oct 30 19:29:22.233213 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:22.233181 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-w4clh_0a1badb7-be53-4b7f-933f-23e2599ab5ba/kube-rbac-proxy/0.log" Oct 30 19:29:22.266895 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:22.266836 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-w4clh_0a1badb7-be53-4b7f-933f-23e2599ab5ba/exporter/0.log" Oct 30 19:29:22.301793 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:22.301761 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-w4clh_0a1badb7-be53-4b7f-933f-23e2599ab5ba/extractor/0.log" Oct 30 19:29:22.410860 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:22.410757 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-69dbdc8f84-kt9h9_7d18009e-009c-40be-9143-e468d1bdfc41/manager/0.log" Oct 30 19:29:22.453745 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:22.453717 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-spxts_d99c9271-c3ea-4f9b-8668-c0348c766af0/server/0.log" Oct 30 19:29:23.244264 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:23.244238 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-k5ljv_5b43299a-0f21-4d69-8167-6f8739c40a9b/proxy/0.log" Oct 30 19:29:23.751807 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:23.751773 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68d996c55722t75_9b0ef8f8-8b08-40e1-8338-fd48c63cfcf7/istio-proxy/0.log" Oct 30 19:29:23.939000 ip-10-0-140-163 kubenswrapper[6960]: E1030 19:29:23.938956 6960 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-w4clh" containerName="extractor" Oct 30 19:29:24.784851 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:24.784823 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-65c8c79996-jjcwx_fa35d05a-b5c7-40b2-ac7a-6d72ebf85a69/kube-auth-proxy/0.log" Oct 30 19:29:24.842121 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:24.842095 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-855f9574cf-dslrt_93b3f3a7-cd85-4996-88fe-7762155b3241/istio-proxy/0.log" Oct 30 19:29:24.917355 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:24.917322 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68d996c55722t75_9b0ef8f8-8b08-40e1-8338-fd48c63cfcf7/istio-proxy/0.log" Oct 30 19:29:24.944709 ip-10-0-140-163 kubenswrapper[6960]: I1030 19:29:24.944679 6960 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-9b4747bc-kdqdg_f6768ac2-1740-48aa-9305-0b3884c1ebf2/istio-proxy/0.log"